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breaking the link between family income and educational achievement, ensuring that children from all 
backgrounds can fulfil their potential and make the most of their talents. 


The EEF aims to raise the attainment of children facing disadvantage by: 
e identifying promising educational innovations that address the needs of disadvantaged 
children in primary and secondary schools in England; 


e evaluating these innovations to extend and secure the evidence on what works and can be 
made to work at scale; and 


e encouraging schools, government, charities, and others to apply evidence and adopt 
innovations found to be effective. 


The EEF was established in 2011 by the Sutton Trust as lead charity in partnership with Impetus Trust 
(now part of Impetus - Private Equity Foundation) and received a founding £125m grant from the 
Department for Education. 


Together, the EEF and Sutton Trust are the government-designated What Works Centre for improving 
education outcomes for school-aged children. 
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Executive Summary 


The project 


The intervention evaluated here is one of two ‘youth social action’ projects jointly funded by the 
Education Endowment Foundation, the U.K. Cabinet Office, the Pears Foundation and the Stone 
Family Foundation. It was delivered by the Youth United Foundation (YUF) and involved uniformed 
youth organisations being established in schools in six areas in the north of England. YUF helped to 
set up new units of The Scout Association, Fire Cadets, Sea Cadets or St John Ambulance in 
participating schools. The number, duration, and frequency of sessions varied: most groups met 
weekly, sessions lasted two hours on average, and the average number of sessions in the academic 
year was 24. Activities were delivered by trained staff from the uniformed youth organisations and in 
some cases also involved adult volunteers, including teachers. 


This project assessed the impact of participation on pupils’ academic attainment, and on wider 
outcomes such as self-confidence and teamwork, using a randomized controlled trial design. The 
wider outcomes are of particular relevance because the participating organisations share core aims of 
inspiring young people to do community work and volunteer, to learn new skills, and to be active 
citizens. Seventy-one schools were randomly allocated to either receive the intervention or not. Of 
7,/81 Year 9 students, 3,377 reported in the initial survey that they would like to take part in the kinds 
of activities offered, and 663 took part in uniformed youth group activities during the 2014/2015 
academic year. A process evaluation was also conducted to collect information about the mode of 
delivery and programme implementation, and feedback from teachers, pupils and parents. 


Key conclusions 


. There is no evidence that the intervention had any benefit in terms of pupils’ academic 
performance. Although the attainment data suggests a small negative impact, the quality of this 
data is too low to draw this conclusion with confidence. The data quality was compromised due 
to changes in national testing. 


. Participation in the intervention is associated with a small improvement in self-reported non- 
attainment outcomes including self-confidence and teamwork. It is possible that these small 
effects are an underestimate due to technical issues regarding the groups of children that were 
compared in the analysis. 


. For pupils eligible for free schools meals, there is no evidence that the intervention had a 
positive impact on academic attainment or self-reported character attributes. Again, the 
attainment data suggests a negative impact, but the quality of this data is too low to draw this 
conclusion with confidence. 


. Almost a quarter of schools did not deliver the intervention due to issues such as lack of teacher 
volunteers, and other schools did not deliver a full programme of activity. Support from senior 
leaders, dedicated space, school staff time, and a dedicated slot in the school day or after 
school were all identified as necessary conditions for successful implementation. 


. Study participants were extremely positive about the intervention and many felt it had a positive 
effect on the behaviour and skills of participating pupils. 


Security rating awarded as part of 
the EEF peer review process 


How secure are the findings? 


The findings related to non-attainment outcomes have medium to high security. The trial was a large, 
well-designed, randomised controlled trial, with schools randomly allocated to either receive the 
intervention or not. The pupils from the intervention schools were similar to those from the comparison 
schools who received no intervention, and relatively few pupils were lost to the analysis due to issues 
such as moving school. However, the non-attainment outcomes were measured using a bespoke 
survey. This survey was well-designed and based on well-established test items, but as it was 
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developed in its current form for the purposes of this particular evaluation, it may not be able to 
provide a standardised measure of non-attainment outcomes. 


The findings related to attainment outcomes have low to medium security. Although the trial was well- 
conducted as described above, it transpired that the Key Stage 3 data which was to be used as an 
outcome measure would not be available. The alternative outcome measures that had to be used 
were based on scores from internal school records and were very weakly related to previous 
standardised test scores. Different schools also used different tests and marking schemes to derive 
the scores. This makes it harder to estimate reliably the size of the intervention’s attainment impact. A 
more reliable estimate of the impact on attainment will be available after these pupils reach the end of 
Key Stage 4 and additional national test data is available. 


What are the findings? 


In general, the analysis to assess the impact of the intervention reveals small ‘effect’ sizes. In the case 
of non-attainment outcomes these are small and positive. This is the case for almost all of the non- 
attainment outcomes measures, as well as the two identified as primary outcomes (teamwork and self- 
confidence). In the case of attainment outcomes they are small and negative. However, for these 
attainment outcomes, the reliability of the finding is undermined somewhat by the quality of the data. 
For reasons beyond the control of the evaluator, the KS3 attainment data collected at the end of the 
trial may not have been consistent across schools. Further evaluation would be required to understand 
whether the intervention did indeed have a negative impact on attainment. Future research could also 
assess the impact of full exposure to the programme over two to three years, to give adequate time for 
the uniformed organisations to establish themselves. 


The process evaluation revealed that pupils, teachers, and parents all had very positive views about 
the intervention, and in general believed that it had a positive impact on the pupils involved. It also 
highlighted a number of factors that had prevented the intervention from being delivered as planned in 
some schools, including a lack of dedicated space and time, a lack of adult volunteers (including 
teachers) to support the uniformed youth group delivery staff, and sometimes a lack of support from 
school senior leaders. Schools should consider these factors when deciding whether to implement a 
similar intervention. 


How much does it cost? 


The costs vary depending on the number of outdoor activities and the size of the groups. Assuming 20 
pupils per school, the cost is estimated at £18 per pupil per year for St John Ambulance, £26 for the 
Sea Cadets, £245 for The Scout Association, and £420 for the Fire Cadets—an average of about 
£180. 


Table 1: Summary table 


Effect Estimated months’ Security 
size progress eNale| 


Outcome Cost rating 


Gain in self-confidence N/A 42488 
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Introduction 


Intervention 


The ‘treatment’ in this trial is not a single intervention as such, but a programme of activities provided 
by four uniformed youth organisations supported by the Youth United Foundation (YUF)—a charity 
supporting a network of voluntary organisations which offer established long term uniformed youth 
programmes. YUF members include Army Cadet Force, Fire Cadets, Girlguiding, JLGB, Royal Air 
Force Air Cadets, Sea Cadets, St John Ambulance, The Boys’ Brigade, The Girls’ Brigade England & 
Wales, The Scout Association, and Volunteer Police Cadets. The YUF acted as a fund manager and 
broker arranging for some of its member organisations to set up units in schools in six areas across 
England (Middlesbrough, Lancashire, Tees Valley, Merseyside/Liverpool, Redcar and Cleveland, and 
Manchester). These uniformed organisations were the St John Ambulance, The Scout Association, 
Sea Cadets and Fire Cadets. Although schools were offered a choice of uniformed groups, some 
schools were not offered the full choice because not all four YUF uniformed organisations were 
available in all of the six areas. The role of the YUF was largely in the recruitment of schools and 
managing grant agreements. It first wrote to schools in the six regions to offer the uniformed group 
programme. Interested schools responded with their choice, and the YUF then put them in touch with 
the uniformed organisation in the region. The schools then ran their recruitment drive to attract pupils. 
Pupils were therefore not given a choice of uniformed group. Individual uniformed groups visited 
schools to recruit pupils. More details are discussed in the section on Participant Selection. 


Each uniformed group had a specific project manager who oversaw the overall delivery of the new 
groups. At the local level, ‘development workers’ appointed by the relevant uniformed group managed 
and organised the activities within schools in their respective areas. The sessions were delivered by 
trained staff from the uniformed youth organisations and in some cases also involved adult volunteers, 
including teachers. Traditionally YUF organisations have opened units in community settings that meet 
predominantly on weekday evenings. This project involved new units based within schools that ran 
during the day or immediately after school. 


All of the uniformed groups followed a regular structured programme of activities, working towards a 
kind of certification. For example, the Fire Cadets worked towards the level 2 BTEC Award in the Fire 
and Rescue Service in the Community, the Sea Cadets followed a curriculum working towards the 
BCU (British Canoeing Union) and the RYA (Royal Yachting Association) Stage 1 qualification, and 
the St John Ambulance followed the First Aider programme to qualify for the Trainee Cadet First Aider 
certificate. The Scout activities were geared towards the collection of Explorer Scouts Badges and 
awards, such as Hill Walking, Performing Arts, Aviation or Public Relations. Some of the activities 
were aligned with the Duke of Edinburgh (DofE) award syllabus (Appendix C4). Each of the four 
organisations had their own syllabus or schemes of work (see Appendices C1, C2 and C3). All 
uniformed groups included a residential camp as part of their programme of activities. 


Although youth social action is not the exclusive focus of the YUF organisations, they all encouraged 
and delivered youth social action activities as part of their programmes. 


The uniformed group activities were delivered weekly (except for one Scout unit which ran once a 
month during the DofE scheme) either within curriculum time or after school. The Scout Association, 
St John Ambulance, and the Fire Cadets held most of their activities within the school grounds, with 
occasional offsite activities, with the exception of one Fire Cadets school where there was no 
appropriate on-school facility. The Sea Cadets activities were organised such that half of the activities 
were held in school and half at local boating sites. All of the uniformed group sessions had an element 
of theory combined with practical exercises in their weekly lessons. These activities were very much in 
line with the aims of the uniformed organisations. Depending on the uniformed group, these activities 
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could be training in fire safety and fire drills, first aid, water sports such as sailing, canoeing or 
kayaking, or rope skills. 


The number of sessions planned differed across uniformed groups: 


e St John Ambulance: 30 weeks, including 12 weeks of first-aid training. 
e Sea Cadets: 33 weeks of activities. 
e Fire Cadets: 38 weeks of activities. 


e Scouts: 30 weeks of activities. 
Background evidence 
Policy and practice context 


A very public debate in the 1990s portrayed young people in the U.K. as disproportionately alienated, 
disaffected, and apathetic (Fogelman 1995, Haste 1996). This led to the establishment of the Youth 
Parliament campaigns by the British Youth Council and the Ministry of Sound, as well as activities by 
Community Service Volunteers and the Carnegie Young People Initiative. Citizenship lessons were 
also introduced in schools in response to these concerns, and now form a part of the standard 
curriculum in most schools in England. 


In recent years such concerns have again been brought to the forefront of parliamentary discussions. 
The current concern is that young people in the U.K. are not sufficiently engaged in civic activities 
such as volunteering and social action, although it is not clear what an appropriate level would be. 
Opportunities for such activities are rarely available in schools (Birdwell, 2013). The Children and 
Young Person boosts to the Home Office Citizenship Survey showed that about half (49%) of young 
people aged 11 to 15 were engaged in some civic activities (Birdwell, 2013). Some studies also 
suggest that young people were put off volunteering and social action by the negative perception 
associated with such activities and also by the lack of opportunities (Ockenden and Stuart 2014). In 
Canada, where participation in youth social action activities is compulsory in some provinces, the level 
of participation is higher (58%). Another reason for the low level of participation was the perception by 
some young people that social action activities were not sufficiently inclusive or open to people from 
different backgrounds (Bradbury and Kay, 2005). Birdwell et a/. (2015a) reported that pupils from fee- 
paying schools were more likely than those in state secondary schools to have the opportunity to take 
part in non-formal learning such as uniformed group activities. 


In 2012, David Cameron, the UK Prime Minister, asked Dame Julia Cleverdon and Amanda Jordan 
OBE to review how the Government, business, voluntary, public and education sectors could work 
together to support young people to engage in social action between the ages of 10 and 20. This 
resulted in the launch in 2013 of Step Up To Serve and the #iwill campaign - a cross-party, cross- 
sector and collaborative campaign to make youth social action a part of life for 10 to 20 year olds in 
the UK. As a consequence of this, the Cabinet Office provided a grant of £5million to 28 organisations 
working with young people across England to deliver a range of youth social action projects in diverse 
settings (Kirkman et al. 2015). 


The new project being evaluated here was co-funded by the Education Endowment Foundation and 
the Cabinet Office. The aim of this project was to provide opportunities to 13- and 14-year-olds in state 
secondary schools in England to participate in youth social action through uniformed group activities. 


Existing evidence 


There is a growing number of youth social action programmes in the U.K. today. Many studies have 
been conducted suggesting positive effects of such programmes on a range of young people’s 
outcomes, such as employability, self-esteem, confidence, and other useful skills. However, few 
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interventions have been evaluated, and where they have, the evaluations were often not rigorous 
enough to assess the impact convincingly. 


For example, almost all of the studies in the review conducted by the Institute for Volunteering 
Research (Ockenden and Stuart, 2014) looking at the impact of volunteering, youth leadership, and 
youth social action, were based on surveys of young people who volunteered to take part in these 
activities and made no comparisons with those who did not. A number of the studies were cited in the 
review as providing evidence of “impact” even though there were no comparison groups or 
counterfactuals and no random allocation of participants in order to control for exogenous factors. Two 
studies—one looking at the impact of Cooperative Street-Games Volunteers (Cooperative Street- 
Games Volunteers, 2014) and one on the impact of Youth Action Network and Centre for Social Action 
(Boeck et al. 2009)—also did not have comparison groups, but reported that youth volunteering and 
social action helped develop ‘social connectedness’ and foster positive behaviours such as empathy, 
cooperation, tolerance, and better understanding of other people. These are strong but unjustified 
causal claims. The findings were based on a survey of the volunteers and case studies of these 
volunteers. 


Even where experimental studies were conducted, the evidence was unclear because of high attrition 
and mixed results. This made it difficult to judge the evidence. One example is a large-scale 
randomised controlled study conducted by seven research teams across the U.S. (Social and 
Character Development Research Consortium, 2010). The study tracked 6,600 pupils in the third 
grade (aged eight to nine) over two years. Attrition was high—31% of the pupils were lost over the two 
years. The study evaluated the impact of school-based Social and Character Development (SACD) 
programmes on 20 school and pupil outcomes related to social and character development. These 
outcomes included self-efficacy, problem behaviour, altruistic behaviour, engagement in learning and 
academic competence, and perceptions of school climate. Data was collected from a combination of 
surveys from pupils, teachers, and primary care givers. The results were mixed: some programmes 
reported beneficial results and some were shown to be detrimental. These apparent effects were seen 
in one year and not replicated in others. The year-by-year analysis showed no evidence of impact of 
the seven SACD programmes (both individually and combined) on pupil’s social and character 
development. Growth curve analysis used to estimate impact over time also reported no ‘significant’ 
effects of the combined programmes on pupil outcomes. Six of the individual results were deemed 
‘significant’, of which two were positive and four were detrimental. There are several limitations to this 
study: there was a large number of missing cases; close to 40% of pupil data was not collected, either 
because the caregiver did not give consent for participation or because pupils were absent; in 
addition, these pupils might have been systematically different to those for whom data was collected. 


A large-scale experimental study by the Cabinet Office in the U.K. evaluated the impact of Youth 
Social Action (YSA) on young people aged 10 to 20 in 73 schools (Kirkman et al. 2016). The 
evaluation consisted of three RCTs and one matched pair trial of YSA interventions provided by four 
providers. All of the programmes included of an element of citizenship activities. Using validated 
questionnaire items as outcomes, the RCTs suggested positive effects of youth participation on young 
people’s work and life skills, such as empathy, problem-solving, cooperation, grit and resilience, sense 
of community, and educational attitudes. In addition to the survey, the study also measured 
observable behaviours: one involved an interview task where pupils’ performance was assessed by 
experienced hirers, and the other was a task where pupils were given four 50-pence pieces and asked 
to decide whether they would keep the money or how much they might donate to charity. The study 
found that compared to their non-participating counterparts, young people who participated in YSA 
expressed greater interest in volunteering activities, but were less willing to donate money to charity. 
Pupils who participated in YSA were more likely to be judged as employable compared to control 
pupils. Effects on attainment outcomes were not available at the time of writing. Although promising, it 
was hard to judge how reliable the evidence was because there was no clear reporting on attrition, 
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school selection criteria, unit of randomisation, randomisation process, or selection of comparator 
group. 


Research on the impact of uniformed group participation has also suggested positive benefits for 
young people. An evaluation of Youth United Foundation projects reported positive effects of 
participation in uniformed group activities on communication, empathy, grit and resilience (Family, 
Kids and Youth, 2015). These findings were based on interview data and case study reports. 
Questionnaire surveys collected responses from young people aged 11 to 18 about their character 
outcomes. A limitation of this study was that treatment and control pupils were not randomly allocated 
and there was a high attrition—40% for the treatment group and 67% for control pupils between pre- 
intervention and post-intervention surveys. 


Evaluation of the impact of Girlguiding (Girlguiding, 2012-2013) and the Duke of Edinburgh award 
suggested that participation increased young people’s resilience and promoted responsible behaviour 
(Duke of Edinburgh, 2010). Almost all of the girls in Girlguiding said that participation in the uniformed 
activity had increased their confidence and leadership skills. Again, these impact evaluations were 
based on young people’s self-report. Young people’s attitudes and changes in behaviour were 
compared over time, but no comparisons were made with similar children not involved in these 
activities. 


One study looked specifically at the impact of participation in the fire and rescue services on young 
people identified as having a range of anti-social and behavioural problems (Ward et al., 2009). The 
study concluded that the structured, disciplined environment and close group work often associated 
with uniformed group activities did have benefits. Although not all completed the course, there were 
reported positive outcomes in terms of behaviour and attitudes from school, home, and peers. The 
young people reported a positive self-concept as a result of participation and several went on to 
become Fire Cadets. There were also reportedly societal impacts in terms of fewer hoax calls or 
deliberate fires started by young people. A decline in the number of offences committed by young 
people in the community was noted, and the number of permanent exclusions from school also 
dropped. 


Surveys of the Combined Cadet Force (CCF) in two Welsh state schools suggested perceived 
improvements in attendance, behaviour and attitudes, and social relations (Glover and Sparks, 2009). 
Again, this study did not compare similar pupils who were not in the CCF so it is difficult to say if the 
pupils would have made similar improvements had they not been in the CCF. Teachers reported that 
pupils were better organised, and had better communication and thinking skills. Since teachers were 
not blind to the intervention, they may have had a biased view. 


An evaluation of cadet forces across the U.K. involving 5,100 cadets from the Combined Cadet Force 
(CCF), Sea Cadet Corps or Royal Marine Cadets, Army Cadet Force (ACF), and Air Training Corps 
(ATC) reported positive effects across a range of outcomes, such as leadership skills, teamwork, self- 
esteem, confidence, and positive attitude (Moon et al., 2010). A similar evaluation was carried out in 
2014 looking into the impact of volunteering in uniformed youth organisations such as The Boys’ 
Brigade, Catholic Guides, The Girls’ Brigade, and the Guide and Scout Associations (Volunteer Now, 
2014). Some of the positive benefits reported included learning new skills, gaining qualifications, 
leadership skills, teamwork, and better communication skills. The findings of these studies were based 
largely on anecdotal reports by participants, volunteers, parents, and teachers. Again, no comparisons 
were made with those who were not involved in the uniformed groups so we cannot be sure if they 
would have made similar improvements had they not been in one of these groups. It is possible that 
pupils who participated in these activities were already likely to have high self-esteem and be more 
academically able and confident, alternatively, the changes observed could have occurred anyway. 
Without proper randomisations it is hard to say if this was the case. 
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Much of the evidence so far is based on participants’ self-reports, or on survey and interview data, and 
uses no independent or validated measures. Although some of these studies claimed positive effects 
on a wide range of outcomes, the research designs used were often poorly suited to isolating the 
impact of an intervention from the many other factors that can contribute to changes in participants’ 
self-evaluation and to changes in more objective outcomes. It is often the case that when participants 
in a programme are asked for their views of the programme they tend to give desirable answers. 


There is currently a lack of robust evaluation of the impact of organised uniformed youth groups. Also, 
none of the studies so far has assessed the impact of youth participation in social action activities on 
academic outcomes. This new evaluation is the first large-scale study conducted in the U.K. to 
evaluate the impact of participation in uniformed group activities on the academic attainment of young 
people and the development of positive wider outcomes using a randomised controlled design. 


Evaluation objectives 
The main research questions are: 


1. What is the impact of access to participation in school-based uniformed youth group activities 
for one academic year on pupils’ attainment in maths and English measured at KS3? 


2. What is the impact of access to participation in school-based uniformed youth group activities 
for one academic year on pupils’ wider outcomes with a special focus on self-confidence and 
teamwork? 


3. What is the impact of access to participation in school-based uniformed youth group activities 
on academic attainment in terms of performance in maths and English at KS3, and wider 
outcomes, for disadvantaged pupils, defined as those eligible for Free School Meals (‘FSM 
pupils’)? 


Project team 


The project was managed by the Youth United Foundation led by Gavin Delf, while individual 
uniformed groups with their regional leads managed and organised the activities within schools in their 
respective regions. The YUF distributed the funding to its members to deliver the project and 
supported them in the recruitment of schools, distributed and managed grants to the four member 
organisations delivering the programme, oversaw the ongoing communication with treatment schools, 
and managed control schools. The evaluation was led by Durham University with the co-operation of a 
team from Leicester University. Professor Stephen Gorard managed the evaluation and had a 
particular focus on the impact evaluation. Dr Beng Huat See managed the project, including designing 
the survey questionnaire and communications with the developer and the schools, and had a 
particular focus on the process evaluation. Dr Nadia Siddiqui, Professor Emma Smith, and Dr Patrick 
White assisted with all aspects of the study, including literature review and preparing the survey items. 


Ethical review 


Ethical approval was sought from, and granted by, the Durham University Ethics Committee. The 
project was conducted in accordance with the School of Education Code of Practice on Research 
Ethics and in line with the British Educational Research Association’s ‘Revised Ethical Guidelines for 
Educational Research’ (2004). These guidelines assured anonymity and confidentiality. No individual 
pupil or school would be identified or identifiable. Schools and individual organisations also obtained 
parental consent for activities (Appendices F and G). 
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Trial registration 


We do not think post hoc registration of the trial is necessary because the protocol has already been 
published, the report will be published in its entirety on the EEF website, and the findings will be in the 
public domain. The reasons for registering a trial are to inform the field that a trial has been conducted, 
and to ensure that all results (both positive and negative) are published and that the trial protocol 
stating the main outcome measures is written before the trial begins to avoid dredging of results or 
changing the main outcomes. Since this trial already conforms to all these requirements, there is no 
need to register the trial. 
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Methods 


Trial design 


This was a simple two-group randomised controlled trial, with randomisation at the school level. 
School-level randomisation was deemed to be the most appropriate approach by the developers and 
funders because it would allow for testing the spill-over effects of the uniformed group on the whole 
cohort within the school, in addition to the impact of actually participating. The plan was to recruit 80 
schools in the north of England with 40 schools receiving the intervention and the other 40 providing a 
control. The intervention started in September 2014 and ended in July 2015. To help reduce post- 
allocation demoralisation, and thus dropout, the control schools were offered an incentive payment of 
£1,500 on completion of the post-test. This was to encourage them to remain in the trial in terms of 
providing the necessary data. 


The plan was for the YUF to offer an average of 20 places to Year 9 pupils in each school. If there 
were more than 20 willing to take part in these schools, the 20 places would be offered to 20 randomly 
selected pupils. Those not selected would form a secondary in-school comparison group. Participation 
was intended to be voluntary. In practice, some schools took a different approach (see the Analysis 
section below). 


Any changes to the protocol are discussed in the relevant sections below. 
Outcome measures 


The aim of this study was to evaluate the impact of participation in uniformed group activities on pupil 
attainment and non-attainment outcomes. 


The two primary attainment outcomes were progress in English and progress in maths (gains between 
KS2 and KS3). 


The two primary non-attainment outcomes were progress in self-confidence and progress in teamwork 
(as measured by survey before and after the intervention). These were pre-selected by the YUF (and 
agreed with the evaluators and the EEF). 


Some longer-term outcomes such as enhanced opportunities for subsequent employment were also 
considered, but would not be measured in the time-scale of this trial. 


The attainment measures 


The attainment outcomes were measured using KS2 English and maths results as the pre-intervention 
attainment measure and KS3 English and maths results as the post-intervention attainment measure. 
The rationale for using KS results was high external validity, and also because it reduces the burden 
of additional tests for both schools and pupils. 


KS2 results were obtained from the National Pupil Database (NPD). KS3 results were also to be 
collected from the NPD. However, after the trial had been planned and was already in progress 
availability of KS3 data through the NPD changed, meaning that national, standardised KS3 scores 
were no longer available. Instead, attainment data recorded by schools at the end of Year 9 was 
collected directly from schools. The data was collected from schools after the end of the trial in 
September 2015 when pupils were in Year 10. Schools used different approaches to measuring 
attainment at the end of Year 9. Results tended to be in two formats. Most schools used their end-of- 
year assessments, which were marked and assessed by teachers, and recorded the results using the 
old KS3 levels for their own use. There was no evidence that these levels were validated. Some used 
GCSE grade schemes (as though the pupil were in Year 10), and one school used their own grade 


scheme. The latter two were all converted to KS3 point scores (from 15 to 53) using conversion tables 
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provided by those schools. While all schools successfully provided data, and this was successfully 
standardised, the process is very likely to weaken the findings for attainment compared to what had 
been planned planned via the NPD. 


These results represent the best possible in the circumstances, and creating them involved 
considerably more work for the evaluators than was originally planned. However, further doubt is cast 
on their validity by the low correlation between the KS2 scores collected from the NPD and the KS3- 
equivalent scores collected as a substitute for the KS3 NPD scores. We would usually expect a 
correlation of about 0.6, whereas for these scores the correlation is only 0.02 for English and 0.01 for 
maths. 


The concerns about the reliability of the outcome measures at KS3 could be addressed by repeating 
the analysis with KS4 data from NDP, when it becomes available, although this would measure the 
medium-term impact of the evaluation, rather than immediate impact. 


Non-attainment measures 


Assessment of the impact of the intervention on young people’s wider outcomes was via a bespoke 
survey instrument developed especially for use in this trial (see Appendix B). The instrument was 
developed by the evaluators in co-operation with the YUF, the EEF, and the Cabinet Office. This 
instrument was piloted in two schools from areas not participating in this trial. 


The instrument comprised basic questions about whether respondents had participated in any 
activities similar to those offered by the YUF, and how keen they were to undertake such activities. 
The pre-test results from these items were to help identify ‘survey volunteers’ in all schools, regardless 
of whether those schools were going to be offered the intervention. 


In addition, the instrument contained a set of single-item questions on a range of wider outcomes 
covering concepts including teamwork, communication, motivation, self-esteem, confidence, 
resilience, civic mindedness, and future intentions (Appendix B). These items were taken from 
validated instruments, some provided by the Office for National Statistics, the Cabinet Office, reviews 
of the literature, prior studies by the evaluators, or professional advice. All have clear audit trails 
leading to their derivation. For example, the item on self-esteem is the one recommended for single- 
item use by Rosenberg (1965). 


The key consideration was that the items were measurable, malleable in individuals, and deemed 
important by stakeholders—either in their own right or because they are linked to behavioural 
outcomes including attendance and participation at school. The instrument was also tested for 
suitability (Such that all pupils could respond with minimal assistance), and as appropriate for the 
reading age of Year 9 pupils. The questionnaire was designed with mostly pre-coded tick-boxes for 
ease of completion. Some items were reverse coded to try and encourage pupils to focus on the 
meaning of each one. Two items were based on short stories (vignettes) in which the socially 
desirable responses were not as clear as in the scaled tick-box questions. 


Administration of survey questionnaire 


The survey questionnaires were individualised with pupils’ names and UPNs printed on them. The 
printing and electronic marking of the questionnaires were commissioned to TRAX, a commercial 
company. The pre-intervention survey was conducted prior to randomisation of schools. The post- 
intervention survey was conducted after the end of the trial in the second and third week of the new 
academic year when pupils were in Year 10. The forms were delivered by couriers to schools two 
weeks before the surveys were to be conducted to allow teachers time to sort them out by classes. 
For the post-surveys, evaluators went to all the schools to supervise the administration of the survey 
and to collect the forms. This helped to ensure that the survey was conducted consistently across all 
schools and also to note any intentional or incidental biases in the way the survey was conducted 
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(since participation was no longer blind). Where the survey could not be conducted within a day, 
several trips were made, and in some cases where evaluators could not be present, forms were 
collected by couriers and delivered to TRAX. Pupils who were absent on the day of the survey 
completed them as soon as they returned and schools were instructed to send these back direct to 
TRAX in pre-paid envelopes. 


Other data 


In addition, data on the background characteristics of pupils in both groups was collected directly from 
schools, and collated, in order to run sub-analyses and to assist potential generalisation of the results 
to other schools and areas. However, for this evaluation, the interest was with FSM-eligible pupils, so 
sub-group analyses were conducted using only FSM-eligible pupils (see section ‘Additional analysis’ 
below). 


Participant selection 
Schools 


School recruitment was via invitation by the Youth United Foundation through their delivery 
organisations in Middlesbrough, Lancashire, Manchester, Cleveland and Redcar, Merseyside/- 
Liverpool, and Tees Valley. The YUF approached 278 schools across these areas via letters, emails, 
telephone calls, and through local authorities and other national and regional partners. Initially 82 
schools indicated an interest and signed the Memorandum of Understanding, but 11 of them failed to 
return initial data and therefore had to be excluded from the trial. The schools were approached on the 
basic of geographical location and selected on a first come, first served basis. Interested schools then 
responded by indicating the uniformed group they would like to offer their pupils (one uniformed group 
per school). Pupils were therefore not given a choice of uniformed group. Schools that indicated 
interest signed a note of memorandum indicating their commitment. The YUF then arranged for the 
uniformed group organisation in the region to contact the lead contact person in the school (who could 
be the deputy head or headteacher) to set up a unit. 


The decision was to recruit only those schools that expressed strong interest, so only 72 rather than 
the planned 80 were eventually recruited because these were the schools that were the keenest. At 
the time of recruitment, two schools were on the point of merging so they were randomised as one 
unit, and the total number of schools became 71. The final 71 schools represented a good mix of 
different school types. Thirty-eight of these schools were randomised to receive the treatment, and the 
remaining 33 formed a ‘business as usual’ control group. More schools were randomised to treatment 
than to control because the YUF wanted to offer the programme to near the number that was initially 
planned, which was 40. This was agreed between the YUF and the EEF. 


Pupils 


Once the school confirmed the uniformed group to be offered, the lead contact person in the school 
ran a recruitment drive to attract pupils. The target pupils were those in Year 9 (aged 13/14). Pupils 
were recruited at the end of Year 8. Schools adopted a range of strategies to recruit pupils. Most 
schools promoted the programme during the school assembly. Some schools invited the respective 
uniformed group personnel to talk about the programme during the assembly. Other schools offered 
taster sessions for pupils to generate interest and also to provide pupils with information about what 
participation in the uniformed group entailed. 


Opt-out consent forms were sent to parents by participating schools (an example is given in Appendix 
D1). If a parent opted out the pupil would be excluded from the trial. In the event, no parents opted out 
of the trial. Consent to participation entails agreement for the use of pupil background, contextual, and 
attainment data. In addition, consent was also sought from parents for participation in outdoor and 
sometimes potentially risky activities (such as handling fire and water activities—see Appendix D2). 
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Media consent was also sought by individual uniformed youth organisations for photographs to be 
used (Appendix D3). 


Changes to protocol 


Although pupil participation in the intervention was meant to be voluntary, in practice schools adopted 
their own systems of recruitment, including simply allocating pupils to the treatment (meaning that 
pupil participation was no longer voluntary). In most schools where the programme was over- 
subscribed, pupils were selected on a first come, first served basis and not randomly selected as 
proposed. Some schools nominated pupils who they thought would most benefit from the intervention. 
In one school, teachers identified those aiming at C to D grades at GCSE to ‘give them the extra 
attention which they would normally not have’. In other schools where the take-up was low, 
recruitment was opened to other year groups. In the case of St John Ambulance, their model was to 
allow all young people interested in participating to do so because they had the capacity to 
accommodate larger groups. (This had implications for the choice of control group: see the Analysis 
section below). 


Sample size 


The study recruited 71 schools and all pupils in these schools—a total 7,781—were asked to complete 
the pre-intervention survey (needed to identify the fair comparator groups). Of these, 4,012 were in the 
treatment schools and 3,769 in the control schools. At the outset, the survey asked pupils if they would 
like to take part in the kind of activities provided by the YUF: 1,733 pupils in the treatment schools and 
1,644 in the control group said they would. These pupils are referred to throughout as ‘survey 
volunteers’. This group includes pupils from both treatment and control schools, including pupils from 
the treatment schools who said they would be willing to take part but who were not actually selected 
for participation. 


Of the 1,733 pupils in the treatment schools who said they would like to take part, 633 pupils are 
known to have participated in the uniformed group activities. A breakdown of numbers by uniformed 
group is given in the table below, and the number of cases in each group at each stage is summarised 
in Figure 1. Three treatment schools were not able to open the Scout unit because of a combination of 
poor uptake by pupils, lack of teacher volunteers, and change of management. Four schools which 
were supposed to be offering St John Ambulance could not get a teacher volunteer to run the unit. 
Nonetheless, all seven of these schools were analysed as being in the intervention group because of 
the intention-to-treat approach, as discussed in the Analysis section below. 


Table 2: Number of Year 9 pupils participating in uniformed activities 


Organisation Offered to schools School participated Pupil participation 
Fire Cadets 11 11 202 
Scouts 8 5 80 
Sea Cadets 8 8 119 
St John Ambulance 11 7 232 
Total 38 31 633 


The 71 schools recruited could be considered a small sample (in terms of degrees of freedom to 
allocate to two groups), but the trial involved non-cognitive outcomes which are generally less 
structurally and socio-economically stratified than attainment, and the effect sizes possible for wider 
outcomes are likely to be higher than for attainment (Gorard and Smith, 2010). The ability to detect 
any impact will also be considerably enhanced by the number of observations (pupils) taken for each 
school estimate, and the likely correlation between pre- and post-test outcome scores. Some 
commentators report a ‘minimal detectable effect size’ at this stage, based on the outcome of a future 
significance test. Significance tests do not work as intended by their users, and anyway could not be 
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used with the kind of data encountered in real-life with missing values and so no standard error (as 
here). Therefore, presenting minimal detectable effect sizes based on flawed analysis and unachieved 
assumptions is an error and must be avoided (Gorard, 2016). 


Randomisation 


In this trial, the sampling and the allocation procedure was at school level. The 71 schools recruited 
were randomly assigned to treatment (n = 38) or control conditions (n = 33) using a pseudo-random 
number generator, after the first survey. One school allocated to treatment indicated—after 
randomisation but before the programme started—that they would like to offer the Duke of Edinburgh 
award instead. They received no YUF intervention, but agreed to remain in the trial. They were still 
treated as an intervention school in the analysis. 


Randomisation was conducted by the lead evaluator prior to achieving any school- and pupil-level 
data in order to reduce any chances of selection bias (using a list of 71 random numbers of which 38 
were even and 33 odd). The randomisation process was observed by another staff member who had 
no vested interest in the programme or evaluation process. 


Analysis 


The two primary attainment outcomes were progress in English and progress in maths (gains between 
KS2 and KS3). 


The two primary non-attainment outcomes were progress in self-confidence and progress in teamwork 
(as measured by survey before and after the intervention). 


Main analysis (presented in Tables 5-8) 


To measure the impact of the intervention on these outcomes, it was necessary to identify which 
groups of pupils should be compared. Although 7,781 pupils were involved in the project in total, 
selecting the group of pupils for analysis was not straightforward. 


Comparing all pupils in treatments schools with all pupils in control schools risked ‘diluting’ the 
observed impact of the intervention because the intention was that only a minority of pupils in each 
school (20 per school) would actually take part in the intervention. Instead, pupils in both treatment 
and control schools were surveyed to identify those who would be willing to take part in the 
intervention (referred to as ‘survey volunteers’). The intention was to compare the pupils in treatment 
schools who participated in activities with similar pupils from control schools who did not. 


In order to create a valid control group, the intention was that in the treatment schools pupils would be 
randomly selected from the group of ‘survey volunteers’ to take part in the intervention, and in the 
control schools, pupils would be randomly selected by the same process for a control group. 


However, in practice, treatment schools did not randomly select participants from the ‘survey 
volunteer’ group: in some schools they took a first come, first served approach, in others teachers 
decided which children should take part. This meant that a fair control group for the intervention 
participants could not be created. 


Therefore it was deemed that the most appropriate comparison was between ‘survey volunteers’ in the 
treatment schools and ‘survey volunteers’ in the control schools. This is a valid comparison which 
allows us to assess whether there was an impact of the intervention among pupils who said they were 
willing to take part in uniformed youth activities. It does not allow us to make conclusions about the 
impact of the intervention on pupils who did not say they would be willing to take part. 


Education Endowment Foundation 16 


Youth Social Action Trials: Youth United 


It is important to note that any observed effect size may be ‘diluted’ because only a subset of the 
treatment school pupils who were included in the main analysis actually took part in the intervention. 


All of these results (in Tables 5 to 8) were expressed as Hedge’s g effect sizes, and the relevant data 
is presented for pre-intervention (initial balance between groups), post-intervention, and gain scores. 


Missing values 


Where relevant values were missing for any reason (such as post-test survey results, or post- 
intervention attainment scores), these cases were omitted from the relevant impact analysis (and are 
used in the sensitivity analysis instead). ‘Creating’ missing values from values that are not missing is 
clearly illogical, can increase bias, and should not be attempted. 


Additional Analysis 


A number of additional analyses were performed using the same outcomes measures of progress on 
English, maths, teamwork, and self-confidence. 


Pupils in the treatment schools who reported participating in the intervention were compared with all 
other pupils in the treatment schools (presented in Tables 9 to 12). This comparison did not have the 
force of a trial because we know there were systematic differences between the two groups. However, 
it is valuable because it allows us to assess the impact of the intervention considering only pupils who 
actually participated. 


The impact of the intervention on FSM-eligible pupils was assessed by repeating the main analysis, 
but only for this subgroup (presented in Tables 13 to 16). In this analysis ‘survey volunteers’ from the 
treatment schools who were eligible for FSM were compared with ‘survey volunteers’ from the control 
schools who were eligible for FSM. 


All pupils from the treatment schools were compared with all pupils from the control schools 
(presented in Tables A1 to A4). The purpose of this analysis was to investigate whether pupil 
outcomes were influenced simply by being in the same school as those who actually participated in 
uniformed activities. 


Further comparisons (in Appendix A, Tables A5 to A13) also looked at wider non-attainment outcomes 
measured by the survey instrument but not pre-selected by the YUF. 


Most of these results were expressed as Hedge’s g effect sizes, and the relevant data is presented for 
pre-intervention (initial balance between groups), post-intervention, and gain scores. The exceptions 
are some additional wider outcomes, such as those presented in Tables A6 to A13, were in the form of 
changes in frequencies, and these are converted into standardised odds ratios. 


A further analysis considers, for the treatment group, correlating the post-test and gain scores for 
attainment and wider outcomes with their known attendance records provided by the uniformed group 
staff who led the weekly sessions. This is not presented as a ‘fair test’, but it gives an indication of the 
impact of dosage. 


The four headline outcomes are also presented as the dependent variables in multiple linear 
regression models, based on entering three predictor variables in two steps (presented in Table 17). 
The first step contains the appropriate pre-test estimate (KS2 maths score for KS3 maths outcome, 
and so on) and a measure of FSM-eligibility. The second step contains the treatment group. 


None of the results are presented in terms of ‘standard errors’ or their derivatives such as significance 
tests or confidence intervals. Such approaches were never designed for use with incomplete samples 
which, by definition, can no longer be random (Glass, 2014), and the approaches do not and cannot 
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answer the research questions (Hunter, 1997; Lipsey et a/. 2012). They are in the process of being 
abolished more generally (Gorard, 2016). 


The presentation of the key effect sizes, along with the RCT design, the number of cases, and the 
level of attrition are sufficient to judge the security of the findings (Gorard, 2015). However, for ease of 
understanding, we combined these factors into one standard number. This is the number of 
counterfactual cases that would be needed to disturb or alter the finding (Gorard and Gorard, 2016). 


In line with usual practice for EEF trials, all of the analysis was done on an intention-to-treat (ITT) 
basis, which means that as far as data is available, all the participants included in the original 
randomisation of a trial are included in the analysis, even if they dropped out of the treatment. In this 
case, this means that all treatment school pupils who were surveyed before and after the intervention 
were included in the analysis, even if their school did not actually deliver any uniformed youth group 
activity. Doing ITT analysis means that if the ‘dosage’ of the intervention is not as high as expected, 
you can expect a ‘muted’ effect size compared to an ITT trial where the dosage was as intended. 


There may be some pupils in the control schools and treatment schools who did not take part in the 
YUF uniformed group programme but were already, or had been previously, members of uniformed 
youth organisations or had been involved in volunteering activities outside school. For this reason, our 
survey asked pupils whether they had been part of a uniformed youth group or been involved in a 
charity or a voluntary group in the last year, and these were checked for balance between the 
treatment groups. It also means that any impact from the treatment is likely to be ‘muted’. 


There is another reason why we might expect a muted effect size: only a fraction of pupils in the 
treatment schools were planned actually to participate, and in practice the number was even smaller 
than planned. The funders wanted whole-school comparisons, and the developers did not have the 
resources to offer activities to each school cohort. Therefore, a compromise was adopted. But it does 
mean that the results are a likely underestimate of any impact at school level. In conjunction with this, 
the number of sessions delivered in eight of the treatment schools was under 20, which was 
considerably low given that this was a one-year intervention. This could also dilute the potential 
impact. 


Process evaluation method 


The process evaluation collected information about the mode of delivery and monitored the 
programme’s implementation. It serves two purposes: one is to check that those randomised to 
treatment receive the treatment as intended and for the required number of sessions in order for 
intervention effects to be realised; second, it provides intermediate indicators to enable us to explain 
why the intervention had the effect that it did. It also enables us to identify the features of successful 
implementation as well as highlighting potential barriers. 


In this evaluation, the methods of data collection and analysis for the process evaluation included: 


e consultation with the programme organiser, that is, the Youth United Foundation manager who 
acted as co-ordinator of the programme, and his colleagues (he co-ordinated the collection of 
data from schools and the uniformed groups); 


e informal interviews with uniformed group staff, teachers, pupils and parents; 
e document analysis of the programme syllabus and activities; 
e semi-structured interviews with programme implementers and teachers via emails; 


e observations of, and participation in, the uniformed group activities; 
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e the collection of data on dosage from the respective uniformed groups from the attendance 
records; and 


e observation visits to assist and monitor the conduct of post-intervention surveys. 


It was originally envisaged that the Scouts and the St John Ambulance would be run by teacher- or 
parent-volunteers who would be trained by the organisations, but this did not happen: at least initially 
some of the uniformed groups were led by staff from the respective uniformed group organisations. 
Therefore, the planned observation of training of teacher volunteers was not carried out. 


The interviews, participant and implementers’ self-reports, and participant observations together 
enable us to capture the contexts within which the programmes were implemented. All these help to 
identify the barriers or facilitators to implementation of the programme. 


Process evaluation observations involved evaluators visiting the sites where the activities took place. 
These included the schools, fire stations (for Fire Cadets), or sailing clubs or boating areas (for Sea 
Cadet activities). For each of the uniformed groups we randomly selected two schools to visit. For 
each school, two visits were arranged, one at the beginning of the trial and one towards the end, in 
order to capture change in pupils’ behaviour and attitude. In total, 16 site visits were made to eight 
schools to observe the programme in operation. During the trial an opportunity also arose for the 
evaluator to attend a Fire Cadets’ graduation ceremony in one of the schools not originally selected for 
the process evaluation. As parents were invited to witness the occasion we took the opportunity to 
obtain the views of parents as well. 


During the field visits ad hoc interviews were conducted with teachers, uniformed youth group delivery 
staff, and pupil participants. Feedback was obtained from participants regarding their perception of the 
impact of the programme and some of the issues encountered. The evaluation team also collected 
detailed syllabuses of the programmes during the fieldwork. 


Besides the field visits, data was also collected directly from the uniformed groups. The St John 
Ambulance, for example, provided case study reports and their own evaluations of pupil participation, 
and the Fire Cadets collected feedback on pupils’ participation. 


The process evaluation data was collated and analysed by evaluators initially blind to the trial 
outcomes and independent of the impact evaluation. The findings of the process evaluation were, 
however, later used to understand and illustrate the outcomes. 
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Timeline 


Table 3: Timeline 


Activity 


April-May 2014 Youth United Foundation started to recruit schools in April through the 
eight delivery organisations in Middlesbrough, Lancashire, Manchester, 
Redcar, Merseyside and Tees Valley. 


May—June 2014 Evaluation team began researching for questionnaire items for the wider 
outcome survey, identifying items that have been validated and tested. A 
pilot of the questionnaire was conducted with two primary schools. A few 
iterations of the questionnaire were made based on feedback from the 
pupils and teachers. The final questionnaire was sent to TRAX, a 
commercial printing company for printing. 


June—July 2014 Survey forms were delivered to schools by courier. 


July—Sept 2014 Survey forms were collected by courier by TRAX for scoring. Schools 
were then informed of randomisation as soon as they completed the 
survey. 


Sept—Oct 2014 Uniformed group organisations went to schools to recruit pupils. In some 
schools, however, teachers identified those pupils they thought would 
benefit from the programme for participation. 


Oct 2014—July 2015 Uniformed groups ran the activities in schools. 
June 2015 Pupil background data updated and school leavers identified. 
Sept 2015 Post-intervention survey administered. 


Oct-Nov 2015 School leavers’ destination schools were contacted and requests were 
sought for pupils to take post-survey. 


Nov—Dec 2015 All schools completed the survey. 
Schools were contacted to request for KS3 data or any attainment record 
for their Year 10 pupils as NPD no longer holds such data. 
Jan—Feb 2016 Survey responses were cleaned and checked for errors. 
Analyses of attainment and survey data. 
Results were synthesised and final report completed. 
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Costs 


The costs for this project were estimated from inputs provided by the respective uniformed 
organisations running the programme in the schools. These were submitted to the YUF manager who 
collated and compiled the data. 


The costs for running the programmes in school are likely to vary depending on the number of outdoor 
activities that can be arranged. They are also different for each uniformed organisations as the kind of 
activities are different. For example, the Scouts conducted most of their sessions onsite, so no 
transport fares are incurred, whereas the Sea Cadets sessions were conducted weekly in nearby 
sailing clubs, which involved transporting pupils from school to the site. The cost per pupil also 
depends on the size of the unit. For the purpose of this project, we work on the assumption of an 
average of 20 pupils per school/unit. Twenty was estimated as the optimum number that could be 
comfortably supported by most uniformed groups. 
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Impact evaluation 


Attrition 


No school dropped out from the trial completely, but a number did not deliver the intervention as 
required. All 71 schools completed and returned the surveys. However, seven treatment schools 
(three Scouts and four St John Ambulance) were not able to deliver the programme due to a 
combination of factors: 


e Three schools were not able to run the scouting activities because of the lack of volunteer 
trainers from The Scout Association to run the programme, or did not start because of a 
combination of poor pupil uptake, change of leadership, and no availability of a teacher 
volunteer. 


e Four schools could not run the St John Ambulance programme because of difficulty in finding 
a teacher volunteer to run the school unit. 


Attrition from the trial 
A total of 64 ‘survey volunteer’ pupils did not complete the post-survey. Some of these pupils had left 
school and their destinations could not be revealed for confidentiality reasons because some were 
transferred to special schools or pupil referral units for emotional and behavioural reasons. Some were 
transferred for medical reasons or had severe learning difficulties. Some were also absent from the 
post-survey due to: 

e long term illness; 

e¢ permanent exclusion; 

e dual registries (where pupils were taught at a different site for part of the time); or 

e off-site apprenticeships. 


There was also some missing data from the pre-test (Figure 1). Two schools provided only the names 
of pupils in the uniformed groups for the pre-test instead of all the Year 9 pupils. One school 
inadvertently shredded the completed survey forms. 
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Figure 1: Loss of relevant cases from allocation to reporting 


Randomised 
(schoal n=71) 


Schools allocated to intervention 
(n=38) 

Pupils in intervention schools 
(n=4,012) 


Schools allocated to control (n=33) 
Pupils in control schools (n=3,769) 
Did not receive allocated 
intervention (School n=0) 


Did not receive allocated 
intervention (school n=7) 

Number of pupils who indicated 
that they would volunteer (n=1,733) 


Number of pupils who indicated that 
they would volunteer (n=1,644) 


Lost to follow-up 

Attainment: (school 
n=0, pupil n=16) n=0, pupil n=14) 
Survey: (school n=0, Survey: (school 
pupil n=14) n=0, pupil n=13) 


Post-test survey 
data collected 
(pupil n=3,996) 


Post-test survey 
data collected 
(pupil n=3,755) 


Lost to follow-up 
Attainment: (school 


Available for follow up 
but missing up but missing KS3 
KS3 data (n=30) data (n=34) 


Available for follow 


Not analysed Analysed Analysed Not analysed 
(school n=0, (school n=38) (school n= 33) (school n=0, 
pupil n=0) Attainment pupil Attainment pupil pupil n=0) 

(n=3,946) (n=3686) 

Survey pupil Survey pupil 

(n=3995) (n=3753) 


Note: This flowchart is for the pre- and post-intervention survey responses. In total, 64 ‘survey volunteer’ pupils are missing KS3 
data (because they moved schools), 30 from the treatment and 34 from control. And 27 ‘survey volunteer’ pupils are missing 
post-intervention surveys (moved schools), 14 from treatment and 13 from control. These represent 2% and 1% of the initial 
pupils respectively. In addition, a few pupils did not provide a valid response to all items in the survey, and one school shredded 
the initial survey. The exact figures appear in each table of the results. 


Pupil characteristics 


The schools allocated to the two treatment groups were similar in many measurable respects (Table 
4). In addition, as Tables 5 to 8 show, the pupils taking part in the study were very well-balanced at the 
outset in terms of reported attainment in maths and English (KS2) and self-reported self-confidence 
and teamwork skills. 


Education Endowment Foundation 23 


Youth Social Action Trials: Youth United 


Table 4: School-level comparison between treatment groups 


WEVar le) (= altciarlatolame [cele e) Ofoy alice) me] celey e) 
Tod aTolo) El (-aV(-1| TN Casita? p) Percentage n/N (missing) Percentage 
(or 1C-Yo Le) aler-1)) 
School Type 
Academy 16/38 (0) 43% 8/33 (0) 24% 
Community 7/38 (0) 16% 8/33 (0) 26% 
Special 1/38 (0) 3% 2/33 (0) 6% 
Foundation 4/38 (0) 11% 8/33 (0) 24% 
Voluntary 10/38 (0) 27% 7/33 (0) 21% 


Ofsted Rating 


Outstanding 2/38(0) 3/33 (0) 9% 

Good 20/38 (0) 16/33 (0) 50% 

Requires improvement 13/38 (0) 8/33 (0) 24% 
Inadequate 3/38 (0) 6/33 (0) 


STed soto) i (=\V-1| n (missing) ram Cualxsi tare) 
(continuous) 


Number of Y9 pupils 


33 (0) 155 (average 
number in control 


schools) 


38 (0) 155 (average 
number in 
intervention 


schools) 


Pupil-level n/N (missing) Percentage TAN (hai Lal )) Percentage 
‘(ors \x=te fo) a(er-1)) 


SEN 
FSM eligible 5740/5740 (0) 
EAL 


5205/5205 (0) 25% 
12% 


Outcomes and analysis 
Main analysis 


As described in the Analysis section above, comparing ‘survey volunteers’ in the treatment schools 
and ‘survey volunteers’ in the control schools allows us to assess whether the intervention had an 
impact. In terms of progress from KS2 to KS3 in English (Table 5) and maths (Table 6), there is no 
evidence of any benefit from the intervention for pupils in treatment schools who reported wanting to 
take part at the outset. The effect sizes are negative (-0.09 for both subjects, computed using the KS3 
post-test scores). As discussed in the outcomes section, these findings must be treated with great 
caution because the KS3 scores were provided by schools in different formats, and many of these 
‘survey volunteer’ pupils did not actually participate in the intervention itself. 
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Table 5: Progress in English, ‘survey volunteer’ pupils 


KS2 KS3 i Post- 
points points test ‘Eff 
ect’ size 


Note: the KS3 scores are as reported by schools. 
Note: The number of standardised counterfactual cases needed to disturb this result would be approximately 93 (which is 29 
more than the number of missing cases), suggesting that the results are fairly stable—meaning that the number of missing 
cases is unlikely to destabilize or alter the findings (Gorard and Gorard, 2016). 


Table 6 — Progress in maths, ‘survey volunteer’ pupils 


KS2 KS3 SD Post- 
points points test ‘Eff 
ect’ size 


pes | oy [oe [| 
, [7s [aos [8s 
‘Overall ea [aoa [sa | 009 
Note: the KS3 scores are as reported by schools. 


Note: The number of standardised counterfactual cases needed to disturb this result would be approximately 139 (which is 75 
more than the number of missing cases). This means that it would take 75 more missing cases to alter the substantive findings, 
suggesting that the number of missing cases here is unlikely to destabilize the results. 


Both groups reported gains in self-confidence and ability to work with others (teamwork), but the gains 
were larger for the pupils in the treatment group. This is true for both outcomes (Tables 7 and 8), with 
effect sizes of +0.1 and +0.04 respectively. Again, because the groups were balanced at the outset it 
makes little difference whether the effect sizes are calculated for the post-test only or for the gain 
scores. 


Table 7: Progress in self-confidence, ‘survey volunteer’ pupils 


Pre- Post- Post- 
survey survey test ‘Eff 
ect’ size 


| EE 
| 4 [20 [as [ae [| 


Note: The number of standardised counterfactual cases needed to disturb this result would be approximately 163 (which is 136 
more than the number of missing cases). 


Table 8: Progress in teamwork, ‘survey volunteer’ pupils 


test ‘Eff 
ect’ size 


Overall 


Note: The number of standardised counterfactual cases needed to disturb this result would be approximately 114 (which is 87 
more than the number of missing cases). 


Additional analysis: all pupils 
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Similar results to those in Tables 5 to 8 appear in an analysis involving all pupils (not just the ‘survey 
volunteers’ who said they would be willing to take part in uniformed youth group activities), as shown 
in Appendix A Tables A1 to A4. Further results also show greater gains for the volunteer pupils in 
treatment schools on almost all aspects of citizenship and attitude, including professional aspiration, 
volunteering, and willingness to help others (Appendix A Table A5). 


Additional analysis: comparing pupils in the treatment schools who actually participated in the 
intervention with all other pupils in the treatment schools 


When we consider only those who reported actually taking up the uniformed group activities (Tables 9 
and 10), the potentially negative attainment result is slightly weakened. In the treatment schools, those 
taking part were well-matched with those not taking part at the outset. At the end, those taking part 
were slightly behind (effect sizes of -0.08 for both subjects, based on post-test only KS3 scores). 


Table 9: Progress in English, treatment school pupils only 
KS2 KS3 Post- 


points points test ‘Eff 


ect’ size 
para [ro [96 | az | 
Non- 27.7 6.5 10.1 8.0 - 


37.9 
participants 


“overt | 70 | a7 | ay | ae | 66 | wos | ei | 008 


Note: the KS3 scores are as reported by schools. 


Table 10: Progress in maths, treatment school pupils only 


KS2 r=) B) KS3 Post- 
points points test ‘Eff 


ect’ size 
Participants | 487 | 281 | 46 | 37.0 


Non- 3,293 28.1 37.7 
participants 


Note: the KS3 scores are as reported by schools. 


For the non-attainment outcomes, considering only those known to have participated and comparing 
them to the other pupils in the treatment schools, there is almost no sign of relative improvement in the 
two key wider outcomes (Tables 11 and 12). 


Table 11: Progress in self-confidence, treatment school pupils only 


Pre- Post- Post- 
UT AAV UT AAV test ‘Eff 
ect’ size 


Participants 286 : : ; 2: 2.0 3.9 
ee et] a Le ee 


participants 
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Table 12: Progress in teamwork, treatment school pupils only 


Post- 
test ‘Eff 
ect’ size 


Participants 


Non- 3,072 
participants 


Further analysis shows that there was a very small but positive correlation between the attendance 
score for each pupil taking part in the intervention activities, and their progress in English (+0.02) and 
maths (+0.06) from KS2 to KS3. And there was a small positive correlation between the attendance 
score for each pupil taking part in the intervention activities, and their gain scores for teamwork and 
self-confidence (both +0.07). 


Additional analysis: FSM-eligible pupils 


The impact of the intervention on pupils eligible for Free School Meals was assessed by repeating the 
main analysis, but only for FSM pupils (presented in Tables 13 to 16). In this analysis, FSM ‘survey 
volunteers’ from the treatment schools were compared with FSM ‘survey volunteers’ from the control 
schools. 


Although these results do not have the force of a trial (because FSM pupils were not randomly 
selected), it is interesting to observe the outcomes based only on those FSM ‘survey volunteers’. The 
negative results for attainment are more severe (Tables 13 and 14). There is no indication from this 
trial that poorer pupils benefitted in terms of attainment just from being in schools which offered 
uniformed youth group programmes. 


Table 13: Progress in English, FSM-eligible 'survey volunteer’ pupils 


KS2 =} B) KS3 Post- 
points points test ‘Eff 
ect’ size 


‘Overs | 709) 76 a9 aS | a | as | ay) Om 


Note: the KS3 scores are as reported by schools. 


Table 14: Progress in maths, FSM-eligible 'survey volunteer’ pupils 


KS2 =D) KS3 Post- 
points points test ‘Eff 
ect’ size 


Note: the KS3 scores are as reported by schools. 


In terms of non-attainment outcomes for FSM-eligible pupils, there is little difference between the 
control and treatment groups (Tables 15 and 16). Any benefit from the intervention for FSM-eligible 
pupils is less than for pupils more generally, and is negative for the teamwork outcome. 
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Table 15: Progress in self-confidence, FSM-eligible ' survey volunteer’ pupils 


Post- Post- 
survey test ‘Eff 
ect’ size 


2 [sa [ora [22 [aa [ose Te 
a [a [7 [2 [ae [aa | 
mE 


Table 16 - Progress in teamwork, FSM-eligible ' survey volunteer’ pupils 


Pre- Post- Post- 
survey survey test ‘Eff 
ect’ size 


a [20 [62 [ar [ou [se [| 
a 
a2 


Results as regression models 


The correlations between the prior scores for the four headline outcomes and their post-intervention 
equivalents are all low (in the range 0.01 to 0.07). For the two wider outcomes of teamwork and self- 
confidence this is perhaps not surprising, and may reflect the difficulty of capturing such concepts via 
a simple questionnaire. The two non-attainment measures are more closely related to each other at 
each test administration (in the range 0.13 to 0.32). However, pre- and post-intervention scores in 
tests for English and maths are usually more strongly correlated than they are here. Here, the English 
and maths scores are linked to each other at each administration (in the range 0.67 to 0.69), but not 
from KS2 to KS3. This suggests that the KS3 scores provided individually by schools following the 
abolition of levels differ from previous practice. This all means that the regression models explaining 
variation in the four outcomes in terms of two predictors (pre-score and FSM-eligibility) are weak 
(Table 17). Despite this, knowing the treatment group for each pupil makes little difference to the 
overall results, especially for attainment. This suggests that the treatment has had little impact (as 
already shown above), and is not responsible for the difference in post-intervention attainment. 


Table 17: Regression results for the four headline outcomes, all ‘survey volunteer’ pupils 
Model KS3 English KS3 maths Teamwork Self-confidence 


R at step 1 0.23 0.08 0.07 
(prior score and 
FSM status) 


R at step 2 : 0.23 
(treatment 
group) 


For completeness, Table 18 shows the coefficients for the three variables in these four regression 
models. 


Table 18: Standardised coefficients for variables used in models listed in Table 17 
WéVarcle)(= KS3 English KS3 maths Teamwork Self-confidence 


Prorscore —0G2) 00k 008 


Fem 020028 
Treatment 005-0080 
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Costs 


The costs for running the programme varied with uniformed youth groups. It depended on the nature 
of the activities, the size of the unit (the larger the size, the more cost-effective), the equipment and 
facilities needed, and whether the delivery staff from the organisations were paid or volunteers. The 
estimated costs for running the units in school are summarised below. 


Scouts (for 20 pupils per unit) Y1 Y2 Y3 


Trainers fees (for 3 trainers) £120 £120 £120 
Equipment £3,500 


Outdoor activities £2,000 £2,000 £2,000 


Membership fees £700 £700 £700 
Uniform £800 £800 £800 
£7,120 | £3,620 | £3,620 

Sea Cadets (for 20 pupils per unit) 


Booking of Area Boat Stations, time 
of paid staff, running cost, hiring of 
equipment and facilities. 


£525 £825 
Fire Cadets (for 17 cadets) Y2 Y3 


£1,060 
Equipment ar eel 

£1,266 

£10,240 

| a eee 
SJA (for 20 pupils per unit) Y2 Y3 


These costs do not include additional miscellaneous expenses (such as residentials and BTEC 
qualifications). 


Approximate average costs per 
pupil per year 


Sea Cadets 
Fire Cadets 
SJA (does not include uniform) 
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Process evaluation results 


Implementation 


The uniformed youth groups were managed and run by different organisations, and so the method of 
delivery was expected to vary considerably. There were variations across schools and uniformed 
youth groups in the nature of instruction and training and in the number of sessions delivered. The 
number of planned sessions in the one-year intervention was reduced in most schools. The Scout 
units, in particular, had far fewer sessions than the other uniformed youth groups. This was primarily 
due to delay in starting up. As the Scouts unit relied on teacher or parent volunteers, the challenges 
faced in enlisting such volunteers led to the delay. One unit could only manage four sessions, another 
only five sessions in total (compared to 30 that was planned). The Sea Cadets and Fire Cadets 
managed to deliver 30 sessions on average, instead of the 33 and 38 planned respectively. All the St 
John Ambulance units completed the 30-week course (including a 12-week first aid course). The 
average across units was 24 sessions (See Appendix I). Common reasons for not meeting the target 
number of sessions were late start-up of the units, cancellations due to bad weather (as in the Sea 
Cadets), and cancellations by schools due to clashes with school-organised activities. 


The duration of each session also varied by school and uniformed youth group, but on average each 
session was two hours long. Offsite activities were occasionally conducted after school and tended to 
last the whole afternoon. These were for practical training where special equipment was needed. For 
example, canoeing, kayaking and sailing activities took place in a local boating area, and operational 
training was taken in the local fire station. 


The following is a summary of the activities conducted in the uniformed youth groups units that we 
observed. 


Fire Cadets 


Fire Cadets sessions aimed to equip young people with skills to interact and respond to potentially 
risky situations in their community. All Fire Cadets sessions were conducted by specially trained 
instructors from the local Fire and Rescue Service. The weekly sessions followed a structured 
timetable in line with the delivery of the level 2 BTEC Award in the Fire and Rescue Service in the 
Community. This included a mixture of operational firefighting activities, such as hose running, 
shipping a standpipe, foam drill, and search and rescue techniques. The Cadets also learnt and 
practiced the standard drills, safety words of command, fire and water safety, first aid, and concepts 
related to the structure and role of the Fire and Rescue Service, as well as personal development 
activities to develop teamwork, leadership, and communication skills. Visits were also arranged for 
Cadets to the Fire Control room and the Marine Rescue Unit. Fire Cadets in Merseyside had the 
opportunity to attend a three-day residential camp. A sample scheme of work is attached in Appendix 
C1. Fire Cadets were issued cadet uniforms and full personal protective equipment during their 
training (see Appendix F). 


At the end of their training Cadets received a completion certificate and a first aid awareness 
certificate. Upon successful completion, Cadets received an externally and internally verified BTEC 
level 2 award in Fire and Rescue Services in the Community. Awards were also given for the most 
improved Fire Cadet and the most outstanding Cadet as encouragement for their participation. These 
awards were presented at a graduation ceremony. 


Sea Cadets 


Sea Cadets provided training on a naval theme in which young people were given an opportunity to 
learn and practise new life skills and skills needed in water. Sea Cadet units were led by trained Sea 
Cadets staff. Like the Fire Cadets, the Sea Cadets followed a pre-planned syllabus of activities. The 
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weekly sessions consisted of 45 minutes of recreational or fun activities (such as football, tennis, 
balloon stampede, tunnel race, or softball) followed by 45 minutes of Sea Cadet activity. The latter 
included basic drills, semaphore (using Sea Cadets flags), first aid, orienteering, seamanship, 
meteorology, camping skills (using the stove, pitching tents), and boating sessions. See Appendix C2 
for summary of the programme. 


In the boating sessions the Cadets were taught boating skills, for example, canoeing, sailing, 
kayaking, and rowing. All Sea Cadets worked towards the BCU (British Canoeing Union) and the RYA 
(Royal Yachting Association) Stage 1 qualification. Wetsuits were loaned to pupils during these 
sessions (See Appendix G). In the winter, the activities included seamanship, customs and traditions 
of the Royal Navy, uniforms, drill, cooking ration packs, bush-craft, climbing, fire and safety, team 
building activities, ranks and rates, and first aid. They also took part in a weekend Outward Bound 
experience where they tried zipwire, campfire, abseiling, problem-solving, bush-craft, laser-zone, 
caving, and Jacobs's ladder. Visits were organised for the Cadets to the local Sea Cadets Unit where 
students got to witness the full range of Sea Cadets activities. They also had the opportunity to attend 
taster lessons on Marine Engineering. 


One group of Sea Cadets attended a residential weekend at a local unit learning various skills like 
stalking and concealment, shelter building, keeping the boat tidy, galley cleaning, and other chores. 
Another group had a residential in North Wales. These outdoor activities and camps were part of the 
planned activities. 


At the end of the course, trophies were awarded for the Best Cadet, Most Improved Cadet, and for all- 
round commitment. 


St John Ambulance 


St John Ambulance is a charity which provides training for skills required in situations of medical 
emergency. The St John Ambulance course consisted of a combination of theory and practical 
sessions as well as physical activities. The course included 12 mandatory 90-minute sessions on First 
Aid skills to qualify for the first aid qualification. These sessions were conducted outside school hours 
by trained St John Ambulance officers. However, in one school the teacher volunteer—who had 
started as the school contact for St John Ambulance (providing support and logistics and attending the 
sessions led by the St John Ambulance trainers)—eventually took over as the trainer himself. He was 
a trained first-aider and was deemed suitable to lead the sessions. Additional staff members who were 
enlisted to help out had little involvement. One soon left because unable to commit to the 1.5 hours 
per week. 


All participating pupils were enrolled as cadets of St John Ambulance, and were given a polo shirt as 
uniform to wear during the weekly sessions. They were also issued with a personal First Aid Kit which 
they could keep. All pupils completed the Cadet First Aider programme to qualify as Trainee Cadet 
First Aiders. This enabled them to support their school at events such as Sports Day. 


The weekly sessions followed a scheme of work where cadets learnt how to treat a range of injuries 
and conditions including recognising heart attacks, asthma, supporting an unconscious breathing 
casualty, CPR, choking, minor and severe bleeding and bandaging, making emergency phone calls, 
casualty communication and care, and fainting. Every session included a combination of physical 
activity, theory, and practical work (for example using CPR mannequins). Cadets also took part in 
Casualty Simulation sessions where they learnt to apply theatre make-up to replicate bruising and 
different types of wounds, including foreign objects. There were a lot of role-plays and hands-on 
activities where pupils got to practice the skills taught. 


As well as attending weekly sessions, some cadets attended a residential trip where they worked 
closely with other cadets from around the Northwest to complete their assessments for Trainee Cadet 
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First Aider. In these sessions cadets learnt more about the organisation and the different activities that 
cadet units offer. 


At the end of the course, pupils’ first aid skills were assessed (Appendix C3). The St John Ambulance 
assessment day involved trainers coming into the school and setting up a first aid scenario. Pupils 
were invited individually to the room where the assessment was being held and required to react to 
the situation, talking through their decisions as they make them. Trainers were present to observe and 
assess their performance. Upon successful completion of the assessment, pupils received a certificate 
stating that they were first-aid qualified. This certificate is valid for three years, after which they are 
required to renew it. 


Scout activities 


The aim of the The Scout Association is to provide an opportunity for learning new skills and teamwork 
in new settings. The Scout units were conducted by both teacher volunteers within the school and 
instructors from The Scout Association. In some schools the teachers volunteered, in others they were 
appointed by the headteacher. In one of the schools we monitored, the sessions were delivered by 
four or five members of staff from The Scout Association once a month, with a teacher committed to 
overseeing the programme throughout. These monthly Scout Association-led sessions were practical 
sessions where pupils were involved in activities such as navigation (map reading), cooking (including 
what food was suitable for expeditions), team-building activities, tent-building, first aid (dealing with 
cuts, bruises, sprains, dehydration and hypothermia). These largely followed the DofE award scheme 
of work (see Appendix C4). In this school, teacher mentors met pupils regularly—formally and 
informally—over the course of the programme to check on progress. In another school the activities 
were conducted by a Scout Association leader because no teacher volunteers were available, but a 
PE teacher was enlisted to provide support subject to availability. The initial few sessions were also 
attended by the deputy head to observe the delivery. 


The weekly sessions were about one to one and a half hours, except in one school where sessions 
ran for two hours once a month during the DofE scheme. The range of activities and the quality of the 
sessions varied between schools. In some schools The Scout Association designed and delivered the 
programme. In one school the programme was developed by the teacher to tailor the activities to feed 
into the Duke of Edinburgh award. Most of the sessions were conducted within the school, for 
example, in the school hall and library. The Scout programme also included outdoor activities. For 
example, all units went to visit a local mountain biking centre (Great Tower Windermere Activity 
Centre). At the centre, the Scouts learnt a range of skills, such climbing and raft-building. 


Volunteering 


Although volunteering and community service was not the aim of this project, a number of the 
uniformed youth groups were able to incorporate volunteering activities into their programme—some 
more so than others. The Fire Cadets, for example, took part in a number of social action events, 
which included fundraising activities (bag-packing at local supermarkets) and participating in a number 
of events to help the local community (Supplying food and helping out at the local foodbank, and 
facilitating bingo events at local old peoples’ homes). They also helped raise money by selling cakes 
donated by local businesses to buy games for a local children’s hospital. Some Cadets organised and 
ran afternoon activity sessions for pupils at a special school. In one school, the Cadets represented 
their schools in the Remembrance Parade supported by Fire Service staff. 


The Sea Cadets picked their own charity project to work with. The St John Ambulance cadets 
volunteered as first-aiders at their individual schools’ sports days as well as becoming involved with a 
range of St John Ambulance promotional activities in their local areas. Some units also volunteered 
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their services coaching children and adults to play cricket and play netball. They also helped out with 
charity work. 


Stability of the unit number 


Although the number of participants in all the units stabilised after the initial first few weeks, the actual 
number that dropped out from the programme was difficult to estimate. This was because some 
pupils, having initially signed up, dropped out of the programme when they found out later (having 
attended a few sessions) that the programme was not for them. Information collected from the 
uniformed youth group staff and the school lead contact person suggested that a number attended the 
first few sessions just to find out more about the programme. Feedback from pupils also suggested 
that some enrolled not knowing what to expect. 


We were also informed that there were pupils who dropped out because of peer pressure. Some 
pupils were excluded by the uniformed youth group staff for bad behaviour, some because the 
activities were not deemed appropriate for them. It was also reported that some pupils withdrew from 
the programme because they were asked to pay for the DofE programme, where this was part of the 
Scouts programme. 


Barriers to implementation 
Some of the barriers to implementation were: 
Leadership and management support 


There were staff changes in 20 of the schools. In some schools, the deputy head and the head of 
school including the teacher in charge left the school in the course of the year leading to a situation 
where no staff member was left in charge of the programme. This situation only became apparent 
when the evaluators tried to contact schools to arrange for the post-intervention survey. 


While support from management was generally good in most of the schools, in three of the eight 
schools we monitored, the uniformed youth group staff (employed by the organisations, not the YUF) 
suggested that management support could have been better. St John Ambulance unit leaders from 
one school said that they felt that throughout the project there was not sufficient support from the 
school management. 


One school reported a lack of commitment from the teaching staff who were assigned to look after the 
programme. For example, the teacher was only present for some of the sessions. There were a 
couple of schools that used teaching assistants and support staff instead of trained teachers. 
Uniformed youth organisation staff reported that the absence of a teacher made it difficult to manage 
challenging behaviours and pupil engagement. Intervention schools were expected to provide a 
teacher to oversee the programme and ensure pupil attendance. Schools are responsible for the 
safety and welfare of their pupils, so it was expected that all sessions, including offsite activities (Such 
as Sailing, fire house training, and overnight camping) would have a teacher present. 


Almost all the barriers below are, in some way, related to the lack of managerial and leadership 
support. 


Staffing 


Recruiting teacher volunteers to oversee the uniformed youth group was found to be quite challenging 
in a number of schools. Four schools were unable to start a unit because they were unable to find a 
teacher volunteer. A number of schools also delayed the start-up of the programme due to the 
difficulty of finding a volunteer teacher. 
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A related issue was the lack of commitment from teachers. Some schools found it difficult to get a 
teacher committed to the project throughout the programme. In four schools there were three changes 
in the teacher in charge. In one school the PE teacher came to the sessions only as and when he was 
available. 


Lack of communication with school 


Lack of communication between the school and the uniformed youth organisation staff was another 
barrier. The latter commented that communication with some schools was particularly challenging 
where there was no dedicated teacher in charge of the programme. 


Reduction in number of sessions 


Although this was a one-year intervention, some units ran the programme for two terms only. This was 
deemed to be insufficient for pupils to have the full experience. The main reasons were delay in the 
setting up of the units, cancellations due to school exams, detention classes, and other after school 
activities. Schools also occasionally cancelled sessions without prior warning. Teachers also 
sometimes excluded pupils for bad behaviour as a punishment. The Scout Association reported that 
clashes with school activities were a barrier to participation. 


Although most of the units managed to cover the planned programme of activities in their scheme of 
work, trainers expressed concern that the contracted timetable meant that they were unable to provide 
the full experience for the pupils. The Sea Cadets, the Fire Cadets, and St John Ambulance 
expressed the need for more time to cover the syllabus in greater detail. Because some units were run 
within curricula time there was a time constraint. This made it difficult to plan for off-site activities like 
water sports or fire house training, so the experience the pupils had was a watered-down version of 
what they would actually receive if the units were run properly with full support from the school and 
teachers. The decision on whether the activities should be held within or outside curriculum time was 
largely made by the schools (in those schools observed by the evaluators). So support from 
management in allowing time after-school for such activities could be helpful. 


The St John Ambulance staff felt that the one and a half to two hour time slot given to them by the 
school was insufficient to cover what would normally be a three-hour module. Many pupils said they 
were overwhelmed by the amount of work they had to cover, but would be happy to stay longer after 
school to finish some of the activities. St John Ambulance suggested a three-week lesson cycle to 
cover topics like team-building, theory and practical sessions (for example, using CPR mannequins). 


Bureaucracy 


School bureaucracy with health and safety requirements meant that planning offsite activities took 
longer than anticipated, and so fewer such sessions could be conducted. A detailed run-down of the 
activities and a risk assessment report had to be submitted weeks in advance before such activities 
could be carried out. This made it difficult to plan outside school activities for the cadets. A visit to the 
Fire Station, for example, was cancelled because the risk assessment report could not be produced in 
time. The Sea Cadets were also unable to conduct as many sessions of water sports as they wanted 
because every activity had to be carefully coordinated with the school. Taking children out of the 
school required weeks of planning ahead. 


Lack of capacity of the uniformed youth organisations 


A few schools were unable to start the Scouts unit for a variety of reported reasons, including getting 
no clear commitment from schools, inadequate capacity of the uniformed youth group to provide 
training, and inability of schools to provide teacher or parent volunteers to run the unit. On the other 
hand, the St John Ambulance units were found to be too large to be managed efficiently: although the 
recommended number of participants was 20, two schools had over 65 pupils enrolled, and five had 
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over 23. Because St John Ambulance units had the capacity to accommodate large numbers, they did 
not set a quota, but schools were unable to release the proportionate number of teachers to supervise 
the units. This led to some disciplinary issues during training. 


Pupil behaviour 


Uniformed youth organisation staff reported that the bad behaviour of some pupils had affected 
training. They explained that since they are dealing with fire safety or, in the case of the Sea Cadets, 
with water, they have to take safety seriously. Discipline is therefore an important element in the 
training. One contributory factor for the bad behaviour could be that some pupils were nominated 
rather than volunteered for participation. In the case of St John Ambulance there was not enough 
teachers on hand to manage the group. Some schools also did not have a committed teacher to 
supervise the pupils, using instead teaching assistants or support staff. 


Lack of facilities 


Another factor that could affect the quality of the activities was the lack of appropriate facilities. 
Fortunately this was very rare, but schools thinking of running uniformed youth groups, such as the 
Fire Cadets, will need to be aware of what is required. One Fire Cadets school, for example, could not 
hold training sessions in the school ground because the drill yard was too small, and there was no 
proper lighting in the winter. The fire hydrant was also not appropriately placed in the school ground. 
In such circumstances, schools have to consider the logistics of transporting cadets and resources to 
nearby fire stations. 


Half of the Sea Cadets activities were conducted outside the school in the local boating area. This 
meant that time had to be factored in when running such units, and where they were conducted in 
school, there was competition with other school activities (Such as basketball and exams) for the use 
of the school hall. Schools have to be prepared to make such provisions for the unit to run efficiently. 


Lack of parental support 


Lack of parental support could be a barrier to implementation for the Scouts. The Scout Association 
informed us that they relied on parental involvement to facilitate and sustain pupil interest. One school 
reported difficulty in getting parental support because of the cultural gap in parental understanding 
about scouting activities. 


Necessary conditions of success of intervention 


Lessons have been learned from this trial which could inform how uniformed youth organisations 
effectively deliver their activities in state secondary schools. 


Management support 


Support from school leadership is essential to ensuring successful implementation. Management 
support is necessary to ensure that dedicated teaching staff is assigned to look after the programme, 
that space, facilities and adequate time are available for training, that support is given to the uniformed 
youth organisations to run the programme, and that priority is given to the programme. Schools with 
strong leadership support were able to deliver the programme successfully, and their pupils are 
perceived to have benefited much from the experience. One Fire Cadets school reported 100% 
retention. Pupil behaviour, teamwork, and the level of enthusiasm were described as ‘outstanding’. 
Enthusiasm of cadets was reported to be exceptional. 


Dedicated personnel at school 


Having dedicated and committed teachers to run or support the programme can help to ensure 
successful implementation. This is particularly so for Scout groups that are dependent on teacher 
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volunteers. Having a teacher volunteer as the Scout leader to lead the troop could encourage 
participation and commitment from pupils. For example, two of the Scout units had an enrolment of 
over 20 pupils (one had 31 pupils) and were able to keep the numbers constant. Both units had strong 
and enthusiastic teachers leading the training. 


Availability of, and access to, proper training facilities 


For effective implementation of uniformed youth activities such as the Fire Cadets and the Sea Cadets 
accessibility to training facilities is essential. All the Fire Cadets schools apart from one were able to 
run the training within the school grounds. This meant that pupils did not have to travel to the fire 
station for training, thus saving time and cost. Conducting the training in the school also gives visibility 
of the uniformed youth groups to other pupils in the school. Pupils felt proud to be seen in their Fire 
Cadet gear. This can help motivate and encourage other pupils to join. 


Having an optimum pupil number per unit 


For successful implementation, there needs to be an optimum number of pupils. If the unit is too large, 
it may be difficult to manage efficiently. On the other hand, if too small it becomes unviable and not 
cost-effective. In this trial, four schools had fewer than ten pupils (see Appendix H). The optimum 
number depends on the capacity of the uniformed youth organisation to support the programme and 
also the availability of teachers within the school to oversee the delivery and accompany pupils on 
offsite activities, such as residential camps. 


Dedicated time 


Having a pre-specified and regular time for the training sessions is essential to successful 
implementation. Where schools were given a dedicated time, and where this time was protected, the 
training was delivered successfully with minimum interruptions. The units were able to complete the 
planned number of sessions. However, where training time was not protected, priority was often given 
to other school activities, potentially causing disruption. These other activities included detention 
classes for pupils. In addition, the school hall—used for some of the training—was often used for other 
extra-curricular activities as well. This can signal to pupils that uniformed youth group activities are 
less important, which can affect the morale of uniformed youth group staff and pupils. 


Perceived outcomes and attractiveness to stakeholders 


In general, the uniformed youth activities were well received by pupils, teachers, and uniformed youth 
organisation staff, as observed by the independent evaluators. The study recorded positive feedback 
and comments from all participants, including parents. The programmes of activity can be considered 
a success to the extent that they achieved the aims in offering opportunities to disadvantaged children 
to learn important life skills and to participate in youth social action and volunteering activities. 
Feedback from stakeholders (parents, teachers, delivery staff, and pupils) suggests that pupils have 
benefited from these experiences and their lives enriched. Of particular benefit were the weekend 
residential stays, organised by the Fire Cadets, the Sea Cadets and St John Ambulance. These were 
described by trainers as providing valuable experiences which these pupils would not otherwise have 
had. Teachers and delivery staff observed positive changes in pupils’ behaviour as a result of the 
experience. There were many anecdotal accounts of improvements in pupil behaviour and skills. We 
could only report a summary of these. 


St John Ambulance 


The St John Ambulance programme was perceived by pupils as fun, enjoyable and useful. The 
programme was rated seven out of a possible ten by all the pupils. The pupils were impressed with the 
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professionalism of the uniformed youth organisation staff and the quality of the training they received. 
They liked the fact that they were learning something useful in real life, skills that they could actually 
put into practice. Pupils talked enthusiastically about the first aid skills they had learnt. And although 
not all had the opportunity to use their skills, some said they have used their skills to teach younger 
pupils. One pupil said: 


‘When this course is finished | was saying to my mum that I don’t want to stop here. | want to 
go on, like learning more, to help myself improve.’ 


Pupils also liked the fact that they had a certificate at the end that they could put on their CV. The 
pupils said they would strongly recommend the course to other pupils. 


Teachers were impressed with the range of activities experienced by the young people and the 
support given to their pupils. Overall, teachers were extremely pleased with the organisation and the 
management of the uniformed youth group. They also liked the fact that the delivery staff had very 
good rapport with the pupils. The teachers were also very impressed with the pupils’ commitment to 
the programme. They explained that the training received by the St John Ambulance pupils had wider 
application for both the pupil and the school. As qualified first aiders pupils can act as the first aid 
representative for the school on sports days and field trips. Some school trips would not be possible 
without the presence of a first-aider. In this way the training they received benefitted both the 
individual and the school. 


Pupils had asked if they could continue with the programme after the trial. The St John Ambulance 
organisation had expressed their interest in running the programme for those schools that had asked 
for it. 


‘If the school decides to continue with the Unit next year the current Unit Leaders would be 
happy to continue running them. However they would like more support from the school 
management so they have more confidence when running the Unit’ (YUF manager). 


The trainers also reported that in general pupils enjoyed their experience. 


‘Overall the cadets thoroughly enjoyed attending the SJA cadet unit and it is clear that each 
cadet has gained an increase in social skill alongside their First Aid skills’ (St John Ambulance 
delivery staff member). 


In terms of skills, St John Ambulance delivery staff also felt that the project had benefited pupils 
enormously. They reported that the cadets had gained skills in teamwork, confidence, and first aid. 


Delivery staff also provided individual case studies to illustrate the learning outcomes of the cadets. 
Most of these related to their first-aid skills. For example, one pupil removed a foreign object from his 
mother’s eye, another made a sling to support his grandmother’s broken arm. One pupil applied a cold 
compress on his mother’s injured foot. Another pupil treated his friend’s nose bleed. One girl helped 
dislodge a piece of chocolate from the throat of her baby niece. Delivery staff also reported that first 
aiders had competently and confidently dealt with minor injuries such as sprains, strains, minor 
bleeds, and minor head injuries. 


Fire Cadets 


The Fire Cadets also reported that they enjoyed the programme, particularly the range of skills that 
they had learnt, such as how to prevent fire and how to stay safe. They also spoke about learning risk 
assessments, life skills, and first aid. They said they had ‘got a lot out of the course’ in the short time 
and had a lot of fun too. 
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Positive feedback was also received from teachers. One headteacher was so impressed with the Fire 
Cadet programme that he had agreed to be a ‘Champion Headteacher’ for the project. He had also 
offered to host a focus group at the school for other headteachers and lead contacts to attend. This is 
what he said: 


‘| just wanted to let you know how positively we are feeling about the Fire Cadet programme. 
The students are fully engaged and enthused and already we are seeing a positive impact in 
terms of confidence and teamwork. Having watched a couple of the sessions | feel that much 
of the success so far is down to your organisation of the activities and the delivery of your Fire 
Fighters who have been excellent.’ 


Teachers, delivery staff, and parents reported marked improvements in behaviour. At the Fire Cadets 
graduation ceremony attended by parents, staff and Fire Cadets personnel, pupils demonstrated their 
rescue skills. The cadets showed confidence as they came up individually to make their own personal 
speech about their experiences. All the parents we spoke to expressed their pride in their children, 
especially when they saw them for the first time in their Fire Cadet gear. They all commented on how 
pleased they were with their children’s development. Trainers and delivery staff also echoed similar 
comments about how the Cadets had grown over the year, from shy cumbersome individuals to 
confident and competent young adults. 


A number of schools indicated that they would very much like the programme to continue after the 
trial. In fact, some Fire Cadets pupils had collectively written to the Fire Service to ask for the 
programme to be continued in their school. 


Scouts 


The Scouts rated enjoyment as top of the list of things they liked about the programme. Among the 
things they enjoyed were activities such as camping and building tents, learning the different types of 
knots, and how to make fires. Interestingly, many boys said they enjoyed the life-skill sessions more 
than the fun sessions. They all enjoyed the outdoor activity at the local mountain biking centre. 


In terms of what they had gained from their experience, the pupils said that the programme had 
helped to ‘keep us engaged in doing something useful after school, otherwise we would just go home 
and sit in front of the TV.’ A number of boys also said that they were enjoying school more now. One 
said he was now more open to new ideas. Teachers also observed that pupils were able to transfer 
the skills learnt to assist them with other projects at school, and as evidence towards other 
qualifications (such as the Duke of Edinburgh award). 


Similarly, the Scout leaders were pleased with the way pupils responded to the training. Parents were 
also very appreciative about the experience that their children had received. 


‘Just wanted to thank you and all the staff who gave their time at the DofE [Scouts] practise 
expedition at the weekend. Tom had a great time and although it was only the practise part of 
the award, he has a great sense of achievement at completing it.’ 


‘Your time and effort is much appreciated, please pass on my thanks to the other members of 
the team also.’ 


Sea Cadets 


The Sea Cadets also reported that they enjoyed the programme, especially the residential weekend at 
Colomendy in Wales. Almost all the pupils said they had learnt a lot from the programme and would 
have no hesitation recommending it to their school mates. All the pupils said they would like to 


continue if the programme was offered the following year. 
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‘lam wanting to join the Royal Marines Cadets at City of Liverpool Sea Cadets. | have already 
been to look around. | enjoy army cadets and sea cadets so Royal Marines cadets is a mix of 
the two. | want a career in the army so think this will help. The Sea Cadets after school club 
has been great!’ 


When asked why they wanted to join the Sea Cadets, one pupil replied: 


‘l think | may fail my GCSE’s so | need something else for the future on my CV and thought 
the Sea Cadets would help me gain qualifications.’ 


Sea Cadets delivery staff observed that pupils had developed in confidence and communication skills. 
The instructors reported that on the whole pupils responded well to the training and improved 
behaviour was apparent. Individual pupils also reported how they had developed in terms of skills and 
character. Examples of pupils’ reflections on their achievements include: 


‘| have got a lot stronger at working in a team and delegating. | enjoyed building a shelter with 
a small team at Colomendy and | have a great partnership with Jermain at our weekly boating 
sessions.’ 


‘During our short time with Sea Cadets, we have sharpened our already existing skills e.g. 
Teamwork and Leadership, while learning new life enhancing and helpful ones, all through the 
activities we have performed.’ 


‘The Sea Cadets has been Great.’ 
‘It has taught me new skills.’ 


‘We as a team have achieved what we came here for, new skills, experience, but most 
importantly... a good time.’ 


‘l enjoyed it a lot.’ 


Delivery staff also reported obvious improvements in pupils’ self-confidence and leadership skills. One 
pupil had gained so much in the short time that he is now helping to run some of the sessions. As 
pupils were awarded for their achievements and attendance, delivery staff and teachers reported that 
this boosted their self-esteem. Improvements in behaviour, co-ordination, teamwork, leadership skills, 
and communication skills were also reported. Teachers reported that a number of pupils who used to 
have detentions were now regularly attending Sea Cadets. The Sea Cadets also expressed interest in 
continuing with the programme after the trial. Our understanding is that at least one school is now able 
to continue to offer their pupils the Sea Cadets programme through YUF. 


The volunteering projects organised by some of the units provided opportunities to practice 
organisational and leadership skills. Teachers and delivery staff reported that pupils demonstrated 
initiative and innovation in planning and organising these activities. The Fire Cadets Team Leader said 
that the experience gave the Cadets a sense of pride and she was clearly touched herself by the 
gestures and all round efforts of the young people. The food bank manager whom the cadets 
approached for help expressed delight with the Cadets’ efforts and their work attitude. 


Formative findings 


The process evaluation feedback suggests a number of factors that could affect the effective 
implementation of such a programme. In a number of schools the number of sessions actually 
delivered was much lower than planned (See Appendix |). One Scout unit met only once a month. This 
was due to late start-up, longer time taken to recruit interested pupils, and difficulties in getting teacher 


and other volunteers to run the unit. The setup and delivery were therefore more condensed and 
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watered-down versions of the usual uniformed youth activities that one would find in such a setting. As 
a result, the experiences of pupils may be different to those received from the more established 
uniformed youth programmes. Despite the condensed timetable, the majority of the units were able to 
cover the essential aspects of the syllabus. The St John Ambulance cadets, for example, completed 
the 12-week first-aid course, the Sea Cadets completed the training necessary for the BCU and RYA 
Stage 1 qualifications, and the Fire Cadets completed the BTEC level 2 award for Fire and Rescue 
Services in the community. 


However, the quality and the intensity of implementation varied between schools and uniformed youth 
groups. The key differences between the more successful and the less successful schools lie in the 
leadership support and commitment of teacher involvement. The schools that completed the greatest 
number of sessions tended to be those where the school leaders took an active interest in the 
programme, where the lead teachers showed passion and commitment to the programme, and where 
there was relative stability among the teaching staff. The schools which had the lowest number of 
sessions were those where there was a lack of commitment by teachers or difficulty in finding teacher 
volunteers to oversee the units (leading to start-up delays), and crucially where management support 
was absent. Some units were run with little knowledge from the school leaders. This was largely due 
to staff changes and no leadership involvement. Some unit leaders felt that throughout the project 
there was not sufficient support from the school management and that this led to poor attendance and 
disciplinary issues among pupils. Some units reported that a number of sessions were cancelled 
(sometimes without warning) for other school activities. Pupils were also taken out of sessions for 
detention classes, indicating the low priority that schools gave to the programme. A number of delivery 
staff commented that it was a challenge working with schools. They felt that strong support and 
commitment from teaching staff would have helped to ensure that a minimum number of hours was 
dedicated to the programme. It is possible that some schools saw the programme as a short-term 
project rather than as part of the wider school curriculum. As such the programme was not given the 
priority and thus the support needed for effective implementation. 


Recommendations 


This section outlines some of the things that could be considered when introducing uniformed youth 
programmes in state secondary schools. It was clear that setting up the units, maintaining them, and 
getting sufficient pupils to take part were all challenging in some settings. Therefore, some of these 
recommendations are suggestions for how these challenges could be met. 


Strong leadership support 


If uniformed youth programmes are to be encouraged in schools, they have to have the support of the 
school leadership, and both school leaders and teachers must believe in the positive benefits of such 
programmes such that priority is given to the programme in terms of time, staff involvement, and use 
of facilities. 


Make participation count 


To encourage participation and commitment, some kind of accreditation could be given for uniformed 
youth activities. For example, a common suggestion by teachers and delivery staff for improving take- 
up and retention was to promote the uniformed youth group as part of the Duke of Edinburgh award. 
The DofE programme is already involved in after-school activities, such as the Explorer Scouts. 
Feedback from teachers suggests that these uniformed youth activities can add to the DofE award 
and also to pupils’ CV, and also that schools are more likely to give a programme attention and priority 
if it leads to accreditation. 


In addition, the most common reason pupils gave for volunteering participation in the uniformed youth 
group activity was that it would be an additional qualification that they could put on their CV, and this 


was a selling point that schools used to recruit pupils. Therefore, a similar system like the Creativity, 
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Action and Service (CAS) component of the International Baccalaureate could be introduced where 
credits could be awarded for uniformed youth group participation and these would count towards the 
GCSE grades. 


Make participation part of the school curricula 


For uniformed youth organisation activities to be successfully implemented, consideration has to be 
given to finding teachers—who are already committed to a range of activities in the school—willing to 
commit the time. Volunteering to help out with the uniformed youth group is at the teachers’ own 
time—over and above their other commitments. One model that has been successfully adopted in 
many independent schools is to structure the timetable to allow for one afternoon for one day a week 
to be devoted to extra-curricular activities. A growing number of academies are already extending the 
school hours and the current government is considering increasing the length of school day to nine 
hours. These extra hours could be used for such activities. If uniformed youth group activities are seen 
as part of the wider school curriculum, it may be easier to get headteacher support and commitment 
from teachers. Teacher involvement in such activities would then be considered as part of their 
teaching responsibility and not as an additional workload. 


More outdoor or hands-on activities 


To encourage participation and enjoyment, more outdoor activities could be organised. This is a very 
common request made by pupils, delivery staff, and teachers. Some arrangements could be made to 
facilitate this as currently the bureaucracy of health and safety requirements meant that such activities 
could not be planned. One option might be that training providers could inform schools at the 
beginning of the year of the number and nature of outdoor activities that will be held. In this way, 
standard risk assessments could be applied in advance. 


Other points suggested by the delivery staff 


e The Sea Cadets would have preferred to run the programme in the local Sea Cadets Unit 
where the facilities and equipment are kept and where pupils could also see what other cadets 
do. This would be more motivating for them. 


e The Scouts suggested having more resources for outdoor activities, budgeting for overnight 
camps at outdoor activity centres, and incentivising pupils by engaging in joint activities with 
explorer Scouts. 


e St John Ambulance teachers suggested having the same delivery staff for all sessions for 
continuity and to allow pupils to build a rapport with the trainers. They suggested having 
trainers who were professional first-aiders, and more opportunities to try out and use the skills 
taught. Better integration of presentations and practical activities was also recommended. 


e The St John Ambulance also suggested having a Youth Development Support Officer, able to 
attend on a weekly basis and support Unit Leaders when delivery staff do not turn up or if they 
have been unable to create plans for a session. 


e Some teachers suggested interviewing pupils to ensure that the ‘right’ people were selected or 
enrolled on the course. 


e Delivery staff said they would like to have better information about the pupil participants 
beforehand so that they could tailor the course. 


Control group activity 


No special programmes of activities were organised for the control schools. To prevent demoralisation 
and dropout from the trial, they were offered an incentive payment of £1,500 on completion, and 
delivery of all post-intervention data. This allowed control schools to use the money to start a new unit 
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in their school, if they wished. To ensure their continued engagement termly newsletters from the YUF 
were sent to these schools. 
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Conclusion 


1. There is no evidence that the intervention had any benefit in terms of pupils’ academic 
performance. Although the attainment data suggests a small negative impact, the quality of this 
data is too low to draw this conclusion with confidence. The data quality was compromised due 
to changes in national testing. 


. Participation in the intervention is associated with a small improvement in self-reported non- 
attainment outcomes including self-confidence and teamwork. It is possible that these small 
effects are an underestimate due to technical issues regarding the groups of children that were 
compared in the analysis. 


. For pupils eligible for free schools meals, there is no evidence that the intervention had a 
positive impact on academic attainment or self-reported character attributes. Again, the 
attainment data suggests a negative impact, but the quality of this data is too low to draw this 
conclusion with confidence. 


. Almost a quarter of schools did not deliver the intervention due to issues such as lack of teacher 
volunteers, and other schools did not deliver a full programme of activity. Support from senior 
leaders, dedicated space, school staff time, and a dedicated slot in the school day or after 
school were all identified as necessary conditions for successful implementation. 


. Study participants were extremely positive about the intervention and many felt it had a positive 
effect on the behaviour and skills of participating pupils. 


Limitations 


This section identifies issues which may have affected the trustworthiness of some aspects of the 
evaluation. 


Identification of target pupils 


The selection of pupils varied across the schools. Those pupils who initially indicated interest may not 
necessarily be those who eventually participated in the funded activities. In many schools there were 
40 to 60 who indicated interest initially but the number who eventually stayed on was much lower. 
Many of these pupils signed up for the taster sessions to find out what the programme had to offer 
before making up their mind. This made it difficult to define attrition (in terms of individual 
participation). Pupils left for a host of reported reasons: lack of interest, clashes with other school 
enrichment activities, detention classes, withdrawn for disruptive behaviour, or in some cases the 
programme was found to be unsuitable for the pupil. In a number of schools, participants were not 
volunteers but selected by teachers for participation (See Methods section on Changes to Protocol). 
To overcome this problem, we asked three questions in the survey to determine those who had 
volunteered participation, those who were selected for participation, and those who would have 
enrolled if they were offered the chance. The latter group was labelled ‘survey volunteers’. 


‘Muted’ effect size 


Given the fact that only a fraction of pupils in the treatment schools were planned actually to 
participate, and in practice the number was even smaller than planned, the estimated impact of the 
intervention is likely to be muted. The funders wanted whole-school comparisons, and the developers 
did not have the resources to offer activities to each school cohort. Therefore, a compromise was 
adopted. But it does mean that the results are a likely underestimate of any impact at school level. In 
conjunction with this, the number of sessions delivered in eight of the treatment schools was under 20, 
which was considerably low given that this was a one-year intervention. This could also dilute the 
potential impact. 
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Validity of performance data 


There are doubts about the comparability of the end of Year 9 data (equivalent to the former KS3 
assessment) provided by some schools. Due to changes in availability of KS3 data through the NPD, 
national, standardised KS3 scores were no longer available. In other words, there was no available 
national measure of pupils’ end of Year 9 performance in English and maths. Collecting data on this 
performance from individual schools was not only time-consuming, but it meant that the evaluation 
was based on whatever schools could provide. Most schools provided the equivalent of KS3 levels or 
points scores. Some used GCSE early grades that could be converted to KS3 scores. With a few 
schools it was not clear whether the scores were current or predicted grades. One school provided 
test scores that were incompletely matched to pupils on the pre-survey list. Another school claimed 
that they were not happy with the data they provided as they did not think it was accurate, but it was 
what they had at the time. This should affect the confidence one has in the attainment results. 


Attrition 


There was a high staff turnover. In 20 of the schools several lead contact teachers were changed in 
the course of the year. New staff members sometimes had no knowledge of the school involvement 
and were not aware of the school’s agreement to provide data or to conduct the survey. 


Pupil turnover was also quite high in a number of secondary schools. At least three schools had over 
50 school leavers in the year. These were always followed up to their new schools, where known, but 
not all of the survey forms sent with pre-paid reply envelopes were returned. A number of pupils were 
also absent for one or other of the surveys despite the two-week window to complete the survey. 
Besides school leavers and absentees, a number of pupils were also missing from the survey either 
due to long-term illness, permanent exclusion, being educated on dual sites, or on apprenticeship 
schemes and educated offsite. One school closed in the middle of the project and its pupils were 
distributed across four to five neighbouring schools. Every effort was made to locate these pupils. 


All of this affected the survey more than the initial attainment data from the NPD (which was based on 
the UPNs of pupils present in the 71 schools at the outset). 


Interpretation 


In terms of attempts to improve attainment, especially for the poorest pupils in England, there is no 
evidence that the intervention had any benefits. It is likely that the abolition of KS3 levels confused the 
attainment outcomes, making them less trustworthy. But it is clear that there is no evidence that 
treatment schools, the ‘survey volunteer’ pupils within them, or even the actual participants improved 
their attainment more than would be expected without the intervention. 


In some ways, this is consistent with the objectives of the uniformed youth organisations, with their 
emphasis on character-building and civic participation. Here, the evidence is stronger, more positive, 
and also consistent with prior work. The headline indicators selected before the trial by the YUF (for 
self-confidence and team work) suggest a small differential improvement for the ‘survey volunteers’ in 
the treatment schools. This is backed up by the same kind of improvements for all pupils in the 
treatment schools—in other words, if there is an effect it is not caused solely by motivation or self- 
selection—and for all other indicators of youth social action, aspiration, and well-being. The same 
improvements appear in the responses to the social action vignettes as in the self-report scales. Given 
that the intervention itself was received by only a minority of pupils per year, only a small average 
result was to be expected. Therefore, it is possible to conclude that there is some evidence that the 
intervention is effective for non-attainment outcomes as portrayed by self-reported results for a wide 
range of non-cognitive outcomes. 


Future research and publications 
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Future research could look at the impact of full participation or exposure to a uniformed youth group 
programme. As this was a one-year intervention, with some units starting late and several sessions 
cancelled, some pupils did not have the full experience of being part of their respective uniformed 
youth organisations. One unit had only four, and another had only five sessions in total. Uniformed 
youth organisations expressed the desire to involve pupils in more outdoor and volunteering activities: 
the shortened programme meant that the courses were not as intensive and broad as they would have 
liked. Future research could look into answering the following questions: 


1. What is the minimum number of sessions required for any impact of participation in a 
uniformed youth group to be realised? 


2. Would continuous participation for three or more years yield stronger results? 


3. What is the long-term impact of school participation in uniformed youth activities in terms of 
employability? 


4. In addition, the concerns about the reliability of the outcome measures at KS3 in this study 
could be addressed by repeating the analysis with KS4 data from the NDP, when it becomes 
available, although this would measure the medium-term impact of the evaluation, rather than 
immediate impact. 


As with all of our previous evaluations, we aim to produce an academic paper from this study. 
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Appendix A: Additional results for the wider outcomes 


Table Al — Progress in English, all pupils 


N KS2 SD KS3 SD Gain SD ‘Effect’ 
points points score size 
Treatment | 3780 Zit 4.7 37.8 6.6 10.1 8.1 - 
Control 3058 257 4.8 38.5 6.7 10.8 8.2 - 
Overall 6838 27,7 4.8 38.1 6.6 10.4 8.1 -0.09 
Note: the KS3 scores are as reported by schools 
Table A2 — Progress in maths, all pupils 
N KS2 SD KS3 SD Gain SD ‘Effect’ 
points points score size 
Treatment | 3780 28.1 4.9 37.6 8.2 9.5 9.5 - 
Control 3058 pe ee | 5.0 38.3 7.0 10.3 9.0 - 
Overall 6838 28.0 4.9 37.9 Ta 55 9.3 -0.09 
Note: the KS3 scores are as reported by schools 
Table A3 — Progress in self-confidence, all pupils 
N Pre- SD Post- SD Gain SD ‘Effect’ 
survey survey score size 
Treatment | 3944 5.8 2.8 7.3 2.1. 1.6 Sie - 
Control 3553 5.0 2.8 Ta 2.0 1.5 3.4 - 
Overall 7497 5.8 2.8 7.3 2.0 1.5 3.4 +0.03 
Table A4 — Progress in teamwork, all pupils 
N Pre- SD Post- SD Gain SD ‘Effect’ 
survey survey score size 
Treatment | 3961 5.2 2.7 6.3 2.6 11 3.5 - 
Control 3544 5.3 2 6.3 2.0 La 3.4 - 
Overall 7505 b.2 27 6.3 2.5 1A 3.5 +0.01 
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sizes for all attitude scale items from the survey instrument. 


Item All pupils ‘Volunteer’ FSM-eligible 
Good at explaining +0.03 +0.06 -0.03 
Like meeting new/0 +0.04 +0.07 
people 

Can work with others | +0.01 +0.07 -0.04 
Can do most things | +0.03 +0.10 +0.04 
Like to complete task | +0.05 +0.09 -0.09 
Make my area better | +0.04 +0.07 +0.17 
Like to be told what | +0.06 +0.04 +012 
to do 

Often afraid to try|0 +0.01 0 
new 

Happy most days +0.01 +0.04 +0.11 
Try to understand | +0.02 +0.06 +0.10 
others 

Know where to get} +0.05 +0.08 +011 
help 


Table A6 — Change in professional aspiration odds ratio from pre- to post-survey, all pupils 


Pre- Pre- ‘non- | Pre-Odds Post- Post- Post- 
‘professional’ | professio | ratio ‘professio | ‘non- Odds 
nal’ nal’ professio | ratio 
nal’ 
Treatment 2121 1834 0.90 2053 1924 O95 
Control 1967 1530 - 1887 1678 
Note: this table compares those listing a professional occupation with all others 


Table A7 — Change in social responsibility odds ratio from pre- to post-survey, all pupils 


Pre- 


‘responsible’ 


Pre- ‘non- 
responsibl 


Pre-Odds 
ratio 


Post- 
‘respon 


Post-Odds 
ratio 


Post- 
‘non- 
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e sible’ responsi 
ble’ 
Treatment 3146 805 1.00 3280 713 L15 
Control 2802 716 - 2853 715 
Table A8 — Change in generosity odds ratio from pre- to post-survey, all pupils 
Pre- Pre- ‘non- | Pre-Odds Post- Post- Post- 
‘generous’ generous’ | ratio ‘generous | ‘non- Odds 
, generous | ratio 
Treatment 1257 2709 0.96 1083 2896 1.05 
Control 1165 2400 - 928 2620 


Table A9 — Change in charitable activity odds ratio from pre- to post-survey, all pupils 


Pre- ‘charity- | Pre- ‘non | Pre-Odds Post- Post- Post- 

active’ charity- | ratio ‘charity- | ‘non Odds 

active’ active’ charity- | ratio 
active’ 

Treatment 1704 2243 0.91 1562 2445 1.02 
Control 1616 1939 - 1380 2206 


Table A10 — Change in professional aspiration odds ratio from 


pre- to post-survey, 


‘volunteers’ 
Pre- Pre- ‘non- | Pre-Odds Post- Post- Post- 
‘professional’ | professio | ratio ‘professio | ‘non- Odds 
nal’ nal’ professio | ratio 
nal’ 
Treatment 1014 695 0.87 994 27 1.04 
Control 1007 600 - 927 706 
Note: this table compares those listing a professional occupation with all others 
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Table A11 — Change in social responsibility odds ratio from pre- to post-survey, ‘volunteers’ 


Pre- Pre- ‘non- | Pre-Odds Post- Post- Post-Odds 
‘responsible’ | responsibl | ratio ‘respon | ‘non- ratio 
e sible’ responsi 
ble’ 
Treatment 1436 276 0.99 1459 265 1.02 
Control 1360 260 - 1376 256 


Table A12 — Change in generosity odds ratio from pre- to post-survey, ‘volunteers’ 


Pre- Pre- ‘non- | Pre-Odds Post- Post- Post- 

‘generous’ generous’ | ratio ‘generous | ‘non- Odds 

, generous | ratio 

Treatment 576 1142 0.94 473 1247 1,09 
Control 5/1 1065 - 417 1200 - 


Table A13 — Change in charitable activity odds ratio from pre- to post-survey, ‘volunteers’ 


Pre- ‘charity- | Pre- ‘non | Pre-Odds Post- Post- Post- 
active’ charity- | ratio ‘charity- | ‘non Odds 
active’ active’ charity- | ratio 
active’ 
Treatment 916 790 1.09 833 896 1.14+ 
Control 900 733 - 737 904 
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Appendix B: The survey instrument 


& & University of 


Durham 


University 


© Leicester 


How do you see your future? 


School URN : 
Name : 
UPN : 
(Please do not complete a survey form that has someone else's Name or UPN printed on it) 
Please ask a teacher to fill in the grid below if your UPN is not printed above 


0 
1 
2 
3 
4 
5 
6 
7 
8 
9 


Instructions 

e This is a survey to find out about your views on things that are important to young people. 
« There are no right or wrong answers to the questions 

« What you say here will not be told to your school or your family 

« Please complete all questions 

e Use only penell to mark the answers. If you make a mistake use the rubber to rub It out 


« Mark the boxes as shown 


Like this: 


= 


NOT like this: 


LL Bx} CO 
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Other 
You Mo aw Greg 
1. Inthe last year, have you been part of any activities or 
groups like those listed below? Ez zz 
DO NOT INCLUDE SPORTS OR SPORTS CLUBS. 


Brownies, Girl Guides, Cubs, Scouts, Rainbows or Beavers 
Children’s University 

Sea Scouts/Sea Cadets, Army Cadet Force, Boys’ Brigade 
e StJohn’s Ambulance, Police Cadets, Fire Cadets 

» Or any other similar uniformed groups 


Other similar activity, not one of the answers above (please write here): 


Other 
Don't (mark here & 
Know write below) 
2. Inthe last year, have you been part of any charity or 
voluntary groups or done any such activities like those Ez Ez 
listed below? 
DO NOT INCLUDE SPORTS OR SPORTS CLUBS. 


» Volunteered to help in a local group, club or place of worship 
e Helped raise money for a charity (e.g. doing a sponsored walk) 
» Done any activities that help other people 


Other similar activity, not one of the answers above (please write here): 


3. Some schools have offered these uniformed groups to their pupils: 
Scouts, St John’s Ambulance, Fire Cadets and Sea Cadets. 


a) If your school offered one of these uniformed groups, would you be a a 
interested in joining? 


b) Were you selected to participate in any of these uniformed groups? | | | <j +¢ | 


c) Did you volunteer to participate in these uniformed groups? Fo o4 of J OF 4 


4. Please read the story below and then draw a line in the box next to the statement that you 


agree with the most: 
Jon notices that there is a lot of litter in the school playing fields. What is the best thing he could do? 
Draw a line 

through ONE box 

e Ignore the mess and take his friends to play in an area that is cleaner. f 4 

e Help to clean up the playing fields. fF 4 

e Complain to his friends that no one cares about the mess. fF 4 

2 
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5. Please read the story below and then draw a line in the box next to the statement that you 
agree with the most: 


Jacinta has difficulty reading and finds it hard to keep up in class. The teacher has to spend a lot of 


time helping Jacinta. Sometimes the other children have to wait for the teacher to stop helping 


Jacinia before helping them. 
Draw a line 
through ONE box 
« Jacinta needs extra help so it is fair that the teacher should spend more time FE 
helping her, even if the other pupils have to wait. 
e Jacinta should work harder to keep up with the rest of the class. fF 4 
» Jacinta should be taught in a separate class. fF 4 


6. Please say how much you agree with the following statements from “not at all true” (0) to 
“completely true” (10). Draw a line through only ONE box on each line. 


Not at all true — Completely true 


0 1 2 3 4 5 6 7 8 9 10 


a. lam good atexplainingmyideasto f| oft 4b 4b db dt de db db db dt 4 
other people 


b. | like meeting new people a a ee ee ee 


c. Icanworkwithsomeonewhohas | jf Jf df dF dt dF ode dkodJe dbo] 
different opinions 


d. I.can do most things if | try be eee a_i 


e. Oncelhavestartedatasklliketo | jf JF dF fF dF Jk dk dk ode de 4 
finish it 


f. Iwanttotryand make mylocalareat 4b off df db dt dk dt dt db dt 4 
a better place 


g. lliketobetoldexactlywhattodo f| Jf ff df df db dk dk Jb db dbo | 


h. lam often afraidtotrynewthings fF Jf if df df odf df dteodbedf dt 4 


i. I feel happy most days a a a | a A |, | 


j. ltrytounderstandotherpeople’s | ft 4b Gb dt dt dt it dt dt dt 4 
problems 


k. Iknowwheretogoforhelpwiha f 4f- 4b db Gk Fk Gk Ge Fe dk dk | 
problem 
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7. Which group contains jobs that are most like the one you would like to do in the future? 


Draw 2 fine 
through ONE box 


e Hairdresser, postman/postwoman, factory worker, cleaner, caretaker, farm worker, fF 4 
waiter/waitress, security guard, fitness instructor. 


e Doctor, scientist, lawyer, vet, teacher, nurse, journalist, artist, 
businessman/businesswoman, musician, computer programmer/web designer 


builder, childminder, car mechanic, clothes designer, sportsperson (6.9. footballer or 
other sports-related jobs) 


E 4 
« Police officer, firefighter, secretary, nursery school teacher, office worker, electrician, [ 4 
e | don't know what job | want fF 4 


« Other (please mark this box and write the job below) | 


If there is a job that you really want to do that is not like any of these, please write it down below : 


Thank you very much for completing this questionnaire. We value your help greatly. 


if you have any further comments about these questions, this survey or Youth United 
then please write them below. 


eseeeeneeed 
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Appendix C: Syllabus 


Appendix C1 


Fire Cadets Syllabus 


SCHOOL PROGRAMME 
SUMMARY OF ACTIVIITES / TERM 1 


Week 1 Collate forms, Staff Intro, Programme Brief, Code of Conduct, Discipline 
Procedure, Instructor contact details, Icebreaker, Ground Rules, Intro to FRS, 
Parade Process & Parade 

Week 2 Visit to Local Fire Station 

Week 3 Risk Assessments, Safety words of Command, Manual Handling, Issue PPE, 
Intro to Equipment 

Week 4 Triangle of fire, Spread of Fire & Effects if Smoke Inhalation, Teambuilding, 
Introduction to Hose Running 

Week 5 Hand signals, Roles within a drill, (Crew formation/Numbering) Cadet 
Handbook, Teambuilding, Standard Drill equipment (45mm hose, branch, 
hydrant, standpipe bar & key) 

Week 6 Bedtime routine & Fire Escape Plans, HSC, Teambuilding, Hydrant 

Week 7* | Bonfire & Fireworks Advice, Standard Drill theory, Drill by numbers, 
Induction handbook 

Week 8 Cadet Quiz, Teambuilding, YH1 Drill (get a branch to work from a hydrant 

Week 9 Issue Uniform, Issue Certificates (Min Pass out at local fire station) 

Week 10 ‘| Fundraising 

Week 11 | Ops Drill, Teambuilding 

Week 12 | Ops Drill, Teambuilding 

Week 13 | Winter Safety, Ops Drills 


NB. Lesson plans for Week 10 to Week 13 have not yet been issued by FSYTA 
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*Depending on the start date of the schools programme — Bonfire/Fireworks safety will be 
incorporated earlier than the planned Week 7 


Course: 

Fire Cadets: BTEC Award 
Level 2 Fire and Rescue 
Services in the Community 


(QCF) 


Course Objectives: 

Provide the opportunity to 
learn about, engage with and 
experience the Fire and 
Rescue Service 

Support young people to make 
a positive contribution to 
society 

Enable young people to gain 
skills, knowledge and positive 
experiences in a fun, safe and 
secure environment 

Support young people to 
develop life skills, increase 
confidence and raise self- 
esteem 

Inspire young people to make 
positive life choices for their 
future. 

Identify the role of the Fire 
and Rescue Service in the 
Community 


Tutor: 

Hayley Rudd 
Kelly Reed 
Mick Jones 
Steve Armstrong 
Jodie Wood 
Mick Bolster 
Dan Smith 


Size of class: 
20 max 


Module/Subject: 
Fire and Rescue Services in the 
Community 


Unit 1 Role of the F&R Service. 
30hrs 


Course Aims: 
Prepare young people for roles and 
responsibilities 


Promote knowledge, skills, 
understanding and attitudes that 
will enable them to _ preform 


effectively, competently and safely 
Provide opportunities to assist and 
encourage 

development 


continuous 


Foster spirit of adventure, develop 
skills and qualities 

Increase awareness of Community 
Safety 

Health 


Develop and 


Awareness 


Safety 


Rescue Services within the 
Community 


Target Group: Duration of Lessons: 

13-14 Years 1.5/2 Hours 

Entry Level: Differentiation: 

BTEC Award Level 2 Fire| VAKR Length of Course: 


37 Weeks 
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Depending on Station Activities, 
during Term 1 


Community Engagement will be scheduled as and when 


Week 1 Responsibilities | Tutor led Fire Kit Observation 
15/09/2014 |of a Fire Cadet, ) Group Class Room Formative 
Parade on and off | work/Discussion | Uniform Assessment 
and Kit issued Pens 
Ground PowerPoint 
Rules/Paperwork 
Week 2 Roles and | Tutor Led PowerPoint Formative 
Responsibilities | Discussion Class Room Assessment 
of the Fire and Hand outs Asng 1 | Hand outs 
Rescue Service - = 1,2,3 
Prevent Protect 
Respond 
Week 3 Introduction to | Tutor Led Fire Kit Formative 
Squad Drills —J Practical Drill Yard Assessment 
Safety Words Hand out Asng 2 | Observation 
Introduction to =6 
Hose Drills Appliance 
Week 4 Team Building: | Tutor Led intro | Fire Kit Formative 
Comms Student Led | Stepping Stones | Assessment 
activity Tyres Observation 
Communication 
Game 
Ball 
Week 5 Unit 1 | Student Led White Observation 
Assignment 1 Presentation Board/Computer | Summative 
Hand Outs Asng | Assessment 
1=4,5 
Folders 
Pens 
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Week 6 Extended Drills —) Tutor Led Appliance Observation 
Add and Replace | Practical Drill Yard Formative 
Ormesby Fire Kit Assessment 
School PD 
Day Fri. 24" 
Oct. 
Half Term 
Week 7 Intro to ASB Tutor Led Class Room Observation 
03/11/2014 Group White Formative 
Discussion Board/Pens Assessment 
Bonfire Hand Out Asng 5 
Night — | Hoax Calls = 18 
Weds. 05/11 Q&A 
Computer/ USB 
Week 8 Unit 1 | Tutor Led Computer Observation 
Assignment 5 Presentation White Summative 
Board/Pens Assessment 
Notes Asng 5 
=19 
Week 9 Recap Skills | Tutor Led Appliance Observation 
session Practical Fire Kit Formative 
Introduction to Drill Yard Assessment 
ladders 
Week 10 Hazard Risk | Tutor Led White Board Hand outs 
Control Measures | Discussion Hand out Asng 2 | Observation 
=7,8 
Week 11 Recap Equipment | Tutor Led Class Room Observation Photo 
— complete hand- | Student Led | White Formative Record 
outs Discussion Board/Pens Assessment needs to be 
Qualities of a Computer taken on 
Team Poster Hand Outs Asng Week 12 
3 =9,10,11,12 
Week 12 Unit 1 | Student Led Appliance Observation 
Assignment 3 Practical Fire Kit Q&A 
Camera Summative 
Assessment 
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Week 13 Unit 1 | Student Led White Observation 
Assignment 2 Presentation Board/Computer | Summative 
Folder Assessment 
Pens 
Half Term 
Half Term 
Week 14 Introduction to | Tutor Led | Class Room Observation 
05/01/2015 | BA- demonstration Computer/USB Formative 
PowerPoint Whiteboard/Pens | Assessment 
Thornaby 
Academy 
PD Day 
Mon. 5 
Jan. 
Week 15 Search & Rescue | Tutor Led Appliance Observation Smoke 
Fire House Practical Fire Kit Formative House to be 
BA Masks Assessment booked at 
Blindfolds LDC 
Week 16 Search & Rescue | Student Led | Blindfolds Observation 
in school - Unit 1 | Practical Masks/Blanks Summative Additional 
Assignment 4 Fire Kit Assessment Staff 
Part 1 Guide Lines Required 
Hand Out Asng 4 Photo 
= 14,15 Record to be 
Camera taken 
Week 17 Recap ASB Tutor Led Class Room Observation 
HFSV Discussion White Board Q&A 
Computer/ USB_ | Formative 
Assessment 
Week 18 Gas Rig Tutor Led LDC Observation LDC Book 
Practical Drill Yard Q&A Gas Rig 
Fire Kit Formative 
Appliance Assessment 
Week 19 Climbing Wall Student Led Climbing Wall Formative 
Practical Assessment 
Half Term 
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Week 20 Assignment Tutor Led | Appliance Summative 
23/02/2015 | Catch up Introduction Fire Kit Assessment 
Unit 1 | Student Led | Drill Yard 
Assignment 4 | Practical Hand Out Asng 4 
Part 2 = 16,17 
Week 21 Heart Start Part 1 | Tutor Led Little Anne Formative 
PowerPoint Assessment 
Computer/USB _ | Discussion 
Hand Outs — First | Q&A 
Aid 
Week 22 Heart Start Part 2 | Tutor Led Little Anne Summative 
PowerPoint Assessment 
Computer/USB Discussion 
Hand Outs — First |) Q&A 
Aid 
Scenarios 
Week 23 Tower Tutor Led Drill Yard Formative LDC to be 
Slip & Pitch -| Practical Appliance Assessment Booked 
Haul Aloft Fire Kit Discussion 
Tower 
Week 24 HP Visit Tutor Led HP Q&A Book HP 
Drill Yard 
Helmets 
Half Term 
Half Term 
Week 25 Fire Awareness Tutor Led Class Room Formative 
13/04/2015 PowerPoint/USB | Assessment 
Q&A 
Week 26 LDC - Crib Fire | Tutor Led Drill Yard Summative Book LDC 
Fire Smoke House Assessment Smoke 
Extinguishers Fire Kit Observation House 
Appliance Q&A 
Week 27 Knots & Lines Tutor Led Knots & Lines Summative 
Practical PowerPoint/UBS | Assessment 
Computer Q&A 
Observation 


Education Endowment Foundation 


61 


Youth Social Action Trials: Youth United 


Week 28 Different Tutor Led Class Room Formative 
Bank Pumping PowerPoint Assessment 
Holiday appliances Computer/USB __| Observation 
Mon. 4" 
May 
Week 29 Pumping from | Tutor Led Drill Yard Formative 
Open Water/ | Practical Appliance Assessment 
Locker Drills Fire Kit Observation 
Q&A 
Week 30 BA Entry Board | Tutor Led Class Room Formative 
PowerPoint Assessment 
Computer/USB | Observation 
Entry Control | Q&A 
Board 
Half Term 
Week 31 Learn & Live Tutor Led Class Room Formative Request 
01/06/2015 Computer/USB _ | Assessment Andy Bright 
Q&A 
Week 32 Foam Drill Tutor Led | Drill Summative 
Practical Yard/Appliance | Assessment 
Fire Kit Observation 
Week 33 1-2-1 Tutor Led Folders Q&A 
Discussion Pens Formative 
Assessment 
Week 34 Crawl Student Led Smoke House Observation Book LDC 
Practical Fire Kit Smoke 
Thornaby House 
Academy 
PD Day 
Mon. 22™ 
Jun. 
Week 35 Pass Out | Student Led Appliance Observation 
Preparation Presentation Fire Kit 
Drill Yard 
Week 36 Create Student Led White Observation 
Presentation Board/Pens 
Computer 
Camera 
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Week 37 


Create 
Presentation 


Student Led 


White 


Board/Pens 


Computer 
Camera 


If available 


switch to 
operational 
week. 


Week 38 


Pass Out 


Tutor 
Introduction 
Student 
Practical 
Presentation 


Led 


Led 
and 


LDC Gym 
Computer 
Appliance 
Fire Kit 

Drill Yard 


Book LDC 
for Pass Out 
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Week | Date Recreational Sea Cadet Activity (45 mins) Teambuilding/ 
Activity (30-45 leadership task 
mins) 

Each week give 1 Royal Navy 
history 
1 3 — 7") Balloon game | General Sea Terms (SP01) If time at end 
Nov RA01-RA13 If have time more general Sea 
Terms (SP26) 
2 10" —JCapture the flag}Seamanship Intro to bends &| If time at end 
14" Nov | RAO1-RA13 Hitches(SP04) 
If more time — Bends & Hitches 
(SP27) 

3 17" JS aside football | First Aid - recovery position | If time at end 

21st Nov | RA01-RA13 (SP10) 

4 24" | Dodgeball RAO1-| Basic Drill — (quick overview of | If time at end 

28" Nov | RA13 reasons for drill SPO5) & (SP06) 
5 1* — 5" | Physical Bosuns calls (SP37) If time at end 
Dec Achievement Communication — Semaphore — use 
mini Sea Cadet flags (no session 
RAO1-RA13 plan) 
6 8 — 12""| Human Chain | First Aid — session 2 — bleeding & | If time at end 
Dec Game RAO1- | shock 
RA13 
7 15" —| Change tag | Bowling If time at end 
19" Dec | RAO1-RA13 
22" —_| CHRISTMAS CHRISTMAS BREAK 
26""Dec | BREAK 
29" — 2°" | CHRISTMAS CHRISTMAS BREAK 
Jan BREAK 
8 5" — 9" | Human knot ]Seamanship — General Rigging | If time at end 
Jan RAO1-RA13 (SP94) 
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a 12" —|Drop touch tag | First Aid — session 3 If time at end 
16" Jan |RAO1-RA13 
10 19" —| Pen the Sheep Royal Marines Cadets tactics If time at end 
23™ Jan 
11 26"  —|Balloon stampede | First aid — session 4 If time at end 
30" Jan |RAO1-RA13 
12 2™ — 6") Tunnel relay | Customs and Traditions If time at end 
Feb RAO01-RA13 
13 9" — 13""| Crab race RAO1- | Leadership Tasks If time at end 
Feb RA13 
16" —|HALF TERM HALF TERM 
20" Feb 
14 23 | Ribble Dibble | Paracord Bracelets 
27" Feb |RAO1-RA13 
15 2™ — 6") Rowing Machines |Rowing Theory and_ rowing | If time at end 
Mar machines 
16 9" — 13""| Recreational PT test If time at end 
Mar Activities 7 
Physical 
Achievement 
RAO1-RA13 
17 16" —|5 a side football | Sailing theory If time at end 
20" Mar | RAO1-RA13 
18 237  —| Practical Seamanship If time at end 
27" Mar | Leadership task 
(SP57) 
19 30" — 3] Recreation Leadership Tasks If time at end 
Apr activity they 
enjoyed RAOI1- 
RA13 
6" — 10" | EASTER EASTER BREAK 
Apr BREAK 
13th =- | EASTER EASTER BREAK 
17th Apr | BREAK 
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Date 


Activity 


20th - 24" Apr 


Boating Qualification 


April 27th - May 1° 


Boating Qualification 


Ath _ gh May 


Boating Qualification 


11th - 15" May 
18th - 22nd May 


Boating Qualification 


L—a3= == ai ee eS a 


Boating Qualification 


25" — 29" May 


HALF TERM 


1st _ 5ih June 


Boating Qualification 


8" _ 12" June 


Boating Qualification 


15th - 19" June 


Boating Qualification 


22nd - 26" June 


Boating Qualification 


June 29th - 3rd July 


Boating Qualification 


Mon 4" July 


After School Provision 
September 2014 to July 2015 


Monday session due to cancelled in May 


22™'_ 26" Sept | Welcome/Introductions/team building activities/Sea Cadet background 


TASTER School 1 
SESSIONS at 

: ; Group 1 Group 2 
boating station 
29" = 3" Oct Sailing/canoe Canoe/sailing 
6'"— 10" Oct Paddlesport/kayak Paddlesport/kayak 
13"-17" Oct | Rowing/canoe Rowing/canoe 
20-24" Oct | Corps knowledge 
27" 31" Oct |HALF TERM 


a7" Nov Seamanship part 1 
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10 — 14" Nov 


Drill part 1 


17" — 21st Nov 


24" _ 28" Nov 


Navigation 


Communications part 1 


1° 5" Dec 


First Aid 


8 _ 12" Dec 


Meteorology 


15" _ 19" Dec 


go"! _ 96" Dec 


ae! 
29" _ 2" Jan 


Recreational activity 


CHRISTMAS BREAK 


a | 


CHRISTMAS BREAK 


5 _ of Jan 


Leadership and teamwork 


12" _ 16" Jan 


Seamanship part 2 


19°93" Jan 


Drill part 2 


26" — 30" Jan 


Communications part 2 


2m _ 6" Feb 


Camp skills 


gt _ 13" Feb 


Recreational activity 


16" — 20" Feb 


HALF TERM 


2371 _ 27" Feb 


Community activity (student volunteering week) 


2m _ 6" Mar 


Marine cadet activity eg stalking 


g _ 73" Mar 


Adventure training 


16" — 20" Mar 


Physical training 


23% _ 97" Mar 


Field craft 


30""— 3 Apr 


Recreational activity (SCHOOLS NOT IN ON FRI 2“? APR) 


6 — 10" Apr 


EASTER BREAK 
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TRAINING School 1 


COURSES at 


' 2 
boating station Croup Group 


13th-17th Apr | EASTER BREAK 


20th - 24" Apr | BCU 1 Star 


April 27th - May | BCU 1 Star 
1 


Ath — 8? May BCU 1 Star 
11th-15" May | BCU 1 Star 


18th - 22nd May | BCU 1 star 


25-29" May |HALF TERM 


ist=5 °°" RYA Stage 1 


8" 12"June | RYA Stage 1 


15th- 19" June | RYA Stage 1 


22nd - 26" June | RYA Stage 1 


June 29th - 3rd| RYA Stage 1 
July 
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Appendix C3 


St John Ambulance syllabus 
PRIMARY SURVEY AND RECOVERY POSITION 


Questions 1 — 10 
CPR 


Questions 11 — 18 
CHOKING 


Questions 19 — 23 
Primary Survey, Recovery Position, CPR and Choking Quiz 


Who is the most important person in first aid? 

What do DR S A B C stand for in the primary survey? 

What three ways can a first aider achieve the ‘R’ of the primary survey? 

How could you ensure a casualty’s tongue is not covering the back of their mouth? 
How long will you spend on the ‘B’ of the primary survey? 

What part of your hand will you use for the ‘C’ of the primary survey? 

What state is the casualty in for you to move them into the recovery position? 

What is the order of the pictures for the recovery position? 

What will you keep checking for after you have moved the casualty into the recovery 
position? 

Although they may not be able to hear you what will you continue to do whilst giving first 
aid? 

What state is the casualty in to deliver CPR? 

What does CPR stand for? 

What two actions make up CPR? 

What is the routine for CPR for an adult? 

What is the routine for CPR for a child, including technique and number of each part? 
What is the routine for CPR for a child, including technique and number of each part? 
How far down the chest do you press down? 

Give three times that you would stop CPR? 

What is the first step if you suspect your friend is choking? 

If it is mild choking what can you suggest? 

What would you do if an adult was choking? 

What would you do if a baby was choking? 

When would it be necessary to call an ambulance? 


Primary Survey, Recovery Position, CPR and Choking Quiz Answers 
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Who is the most important person in first aid? 
You as the first aider 


What do D R S A BC stand for in the’ primary — survey? 
Danger, Response, Stop and Shout for Help, Airways, Breathing, Circulation/ Bleeding/ CPR 

What three ways can a first aider achieve the ‘R’ of the primary survey? 

Talk, Command, Tap 

How could you ensure a casualty’s tongue is not covering the back of their mouth? 
Open the airway 

How long will you spend on _ the ‘B’~ of the primary — survey? 
No more than 10 seconds 

What part of your hand will you use for the ‘C’ of the primary survey? 
Back of your hand 

What state is the casualty in for you to move them into the recovery position? 
Unconscious but breathing 

What is the order of the pictures for the recovery position? 
D, E, A, B, F, C 

What will you keep checking for after you have moved the casualty into the recovery 
position? 

That they are breathing 

Although they may not be able to hear you what will you continue to do whilst giving first 
aid? 

Talk/communicate/reassure 


What state is the casualty in to deliver CPR? 
Unconscious, not breathing 

What does CPR stand for? 
Cardiopulmonary Resuscitation 

What two actions make up CPR? 
Chest Compressions and Rescue Breaths 

What is the routine for CPR for an adult? 


30 chest compressions to two rescue breaths 

What is the routine for CPR for a child, including technique and number of each part? 
5 rescue breaths, 30 chest compressions, 2 rescue breaths, 30 chest compressions etc with 
one hand 

What is the routine for CPR for a child, including technique and number of each part 
5rescue breaths, 30 chest compressions, 2 rescue breaths, 30 chest compressions etc with two 
fingers 


How far down the chest do you press down? 
1/3 
Give three times that you would stop CPR? 


The casualty starts breathing, the paramedics arrive and tell you you can switch with them, 
you become too tired. 

What is the first step if you suspect your friend is choking? 
Ask them if they are choking 
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If it is mild choking what can you suggest 
Coughing it out 
What would you do if an adult was choking? 


Ask if they are choking, up to 5 back blows followed by up to 5 abdominal thrusts checking to 
see if it has come out between each out. 

What would you do i a baby was choking? 
Supporting the head, put the baby on one knee (while you sit down) and up to 5 back blows 
then on the other knee up to 5 abdominal thrusts with two fingers, checking between each. 
When would it be necessary to call an ambulance? 

After you have done the cycle of 10 up to three times, if you have had to use abdominal thrusts 
or they fall unconscious. 


A B 
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Appendix C4 


Duke of Edinburgh — The year ahead 2015 


Bronze 


Date & Time Session and Venue 
Monday 27" January 3.15 — 4.30 1** Meeting [‘Viecet at PE) 


Tuesday 4" February 11.50 —-12.35 2™ Meeting [!Vieet at PE) — Pupils to be put into groups 
IT session 1 (meet in library) 
Monday 10" March 3.15 — 4.30 IT session 2 (meet in library) 


Wednesday 19'* March 3.15 — 4.30 Practical session 1— Map Reading & Compass bearings 
Tuesday 1* April 3.15 — 4.30 Practical session 2 — Camp Craft (Tents) 


Thursday 1** May 11.50 — 12.35 IT session 3 (meet in library) 


Tuesday 3 May 3.15 —5.00 Practical session 3 - Expedition preparation (kit) 


Wednesday 20" May 3.30-4.30 


4.30 


Wednesday 24' June 3.30-4.30 Map and navigation 


Assessment Expedition 
ICT post expedition feedback 
Tuesday 15' July 3.15 — 4.30 IT Session 4 (meet in library) 
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Appendix D: Consent forms 


Appendix D1 


Sample of opt-out consent form 


Prof Stephen Gorard 
Lead Evaluator 
Durham University 
School of Education 
Leazes Road 


DH11TA 


Dear Parent, 


Your child’s school is participating in the Youth United Foundation Social Action Trial which is being 
evaluated by Durham University. The project is funded by the Education Endowment Foundation (part 
of the Department for Education). The aim of the project is to help identify effective strategies that will 
raise the attainment of disadvantaged pupils. To do this, we will need to match the data collected from 
the project with the National Pupil Database. We will not use your child’s name or the name of the 
school in any reporting arising from the research. 


If you do not wish for us to use your child’s data, you can opt out by signing below and return the form 
to your child’s form teacher. 


Your name: 


Signature: Date: 


If you have any questions about this project, please do not hesitate to contact me at: 


Email: s.a.c.gorard@durham.ac.uk 
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Appendix D2 


Activity consent form 


YE) 


FIRE CADETS ACTIVITY CONSENT FORM 


Parental/Carer Consent to Activities 


Date of Birth 

mate ee 
fcuventage |_| 

Address 

a 


Young people attending the Fire Cadets scheme will be involved in a number of activities throughout the 
duration of the programme. Instructors will supervise these activities. 


The general areas of activity is outlined below. Please study the information, and complete the last column 
confirming if you give your consent for the young person named above to partake in each activity. 


es/No 
Squad drills Marching and understanding words of command 
ancwack Running out hose and getting water to work on the 
ground floor and up to the 4" floor of a drill tower 
Ladder work Climbing up and down ladder to first floor level only 


, , Getting on and off fire engines, removing and 
Using fire appliances : ‘ 
replacing equipment 


ia of self, recovery position and calling for po 
Demonstration and assisted use of equipment that 
Road traffic accidents fire fighters use at road traffic accidents and 
understanding of risks of driving 
Hygiene — personal and food Assisting in the preparation of food and washing up ee 
Travelling to and from other venues via a minibus P| 
Working on brigade premises ‘eae on and around a fire station, understanding P| 
Educational sessions delivered by guest speakers to 
Visitor Talks cover topics such as substance misuse, healthy 
eating, sexual health, relationships and road safe 
Wideos maeee Videos/images of road traffic accidents and actual 
9 fires that some people may find disturbing. 


1 
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Appendix D3 


Media consent form 


YE 


FIRE CADETS MEDIA CONSENT FORM 


(For still images, moving images and audio material for CHILDREN) 


We (delete as appropriate), the person with 


parental responsibility of: Age 


give the Fire and Rescue Service permission to use any still and/or moving image 
being video footage, photographs and/or frames and/or audio footage depicting my child 
named above for any of the following uses: 


e Fire Cadets campaigns incorporating video tapes, audio CDs, CD ROMs, DVD's and 
other similar communications and data storage media yet to be invented; 


e Television advertisements, radio advertisements, magazine advertisements, leaflets, 
information packs, flyers, parenting advice publications, the Fire and Rescue Service 
website, the FSYTA website or any website run by the Fire and Rescue Service, 
including social networking as well as any other suitable publicity purposes; 


e Other fundraising and promotional materials, educational materials, research 
materials, lecture outlines, materials required for teaching purposes, for reference in 
the Fire and Rescue Service or FSYTA video and photographic library. 


Is there anything about the child’s history, which you may think we should know about 
before a moving or still image is taken? 


The above consents will apply throughout the world and for an indefinite period. 


Signed: Date: 
Signed: Date: 
Address: 

Postcode: Tel: 
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Appendix E: Cadets collecting for charity 
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Appendix G: Sea Cadets in session 
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School name Organisation Number of pupils 
School 1 Fire 20 

School 2 Fire 4 

School 3 Fire 13 

School 4 Fire 17 

School 5 Fire 20 

School 6 Fire 21 

School 7 Fire 16 

School 8 Fire 11 

School 9 Fire 20 

School 10 Fire 21 

School 11 Fire L 

School 12 Fire 22 

School 13 Scouts Not offered 
School 14 Scouts Not offered 
School 15 Scouts Not offered 
School 16 Scouts Not offered 
School 17 Scouts Not offered 
School 18 Scouts 2) 

School 19 Scouts V7 

School 20 Scouts 23 

School 21 Scouts 31 

School 22 Sea cadets oS 

School 23 Sea cadets 14 

School 24 Sea cadets 16 

School 25 Sea cadets 35 

School 26 Sea cadets 10 
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School 27 Sea cadets 21 

School 28 Sea cadets 4 

School 29 SJA 39 

School 30 SJA 70 

School 31 SJA 65 

School 32 SJA 18 

School 33 SJA 18 

School 34 SJA 14 

School 35 SJA 8 

School 36 SJA Not offered 
School 37 SJA Not offered 
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Appendix I: Number of sessions delivered by school & 
uniformed youth group 


(not all the schools are represented here because we did not receive data from all the units) 


School Organisation Sessions 
School 1 Fire 28 
School 2 SJA 25 
School 3 Scouts 17 
School 4 Sea 19 
School 5 Fire 26 
School 6 Fire 29 
School 7 Fire 31 
School 8 Sea 20 
School 9 SJA 29 
School 10 Sea 21 
School 11 SJA 18 
School 12 Sea 20 
School 13 Sea 30 
School 14 Fire 30 
School 15 Sea 30 
School 16 Fire 30 
School 17 Fire 32 
School 18 Sea 29 
School 19 Fire 30 
School 20 Scouts 19 
School 21 Scouts 17 
School 22 Sea 30 
School 23 Scouts 4 
School 24 Fire 30 
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School 25 Fire 30 
School 26 Sea 19 
School 27 Fire 30 
School 28 SJA 21 
School 29 Scouts 5 
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Appendix J: Cost calculation 


Scouts 
Estimated cost per pupil per year is: £206 


Item Per year 
Trainers’ fees £120 per adult over 3 years | Approx. £120 per unit for 3 
trainers 


Equipment/training kit Start up grant of £3,500 for | £500 per year 
3-5 years and then £500 
year then on 


Outdoor activities £100 per pupil per year £100 per pupil 
Membership fees £35 per pupil per year £35 per pupil 
Uniform for new | £40 (one-off fee) £40 per pupil 
members 

Sea Cadets 


Costs of delivery for the Sea Cadets include the use of the Area boat stations, time and 
resources of paid staff and the running cost, equipment and facilities. The total was estimated 
to be £525 per pupil per year. Of course, this will be higher if the units were smaller. 


St John Ambulance 
Cost per pupil is £17.50 for 20 pupils 


It is estimated that it would cost £350 a year to set up a SJA unit for 20 pupils in the 
school. This is based on running the activities outside lesson time, and so does not 
incur teacher costs. It is also assumed that the unit is run in the school and so no 
hiring of venues or extra facilities required. SJA is a voluntary organisation and their 
staff are volunteers. Presumably the cost is for First-Aid kits for pupils and 
demonstration props such as training mannequins. 


Fire Cadets 


Estimated cost per pupil per year is £800.54 


Based on average of 14 cadets and 2 staff members 
CADET UNIFORM & PPE 3369.1 
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DAY TO DAY RESOURCES (photocopying, 
refreshments, bags to store fire kits, end of term buffet, 
passing out parade buffet, certificates, stationery, 
certificates, postage etc) 1059.72 


EQUIPMENT RESOURCES (hose, foam_ branches 
handcontrol Akron branches, standpipe, inline inductor 2658.14 


TRAINING (behaviour management, safeguarding, drill 
yard supervision) 1266.25 


STAFFING COSTS (X2 Youth Advocates - based on 5hrs 
per cadet night (2hrs delivery, Lhr travel to/from cadet unit 
with resources / 1hr resources preparation / 1hr admin 
x36 school week year 5256 
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Appendix K: Padlock rating 


7" July 2016 Complete by Elena Rosa Brown 


i.Design: pee 3. Attrition: 4. Balance: Siamese 


What is the What is the Adjustment 
minimum 


validity: 


Adjustment to 


uality of the [-\7-1 Me) mel ge) to rating to 
q Y detectable P é account for issues 


design of the folU) ai ge) sem agt= account for 


aluation? lead lad evaluation? balance? bik 
evaluation? valuation? nce? 
start? Tale=igele-1e-halel aig 
Evaluation design Implementation Analysis and interpretation 


Attainment outcomes 


Reville | 1. Design 3. Attrition 4. Balance 5. Threats to 
validity 


No threats to validity 


Fair and clear experimental 
design (RCT, RDD) < 20% 


< 0.3 
38 Well-matched comparison 
(quasi-experiment) <0.4 < 30% 


Matched comparison 


(quasi-experiment) < 40% 


> 50% pial a Significant threats 


pel 
Comparison group with 
poor or no matching < 0.6 
| ear 


The final security rating for this trial is 2 @ for attainment outcomes. 
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Non-cognitive skills outcomes 


Rating 1. Design a 3. Attrition 4. Balance 5. Threats to 


Fair and clear experimental 
design (RCT, RDD) < 0.3 


WEL Celia 


No threats to validity 


Well-matched comparison 

(quasi-experiment) < 0.4 

Matched comparison 

(quasi-experiment) <0.5 < 40% 

Comparison group with 

poor or no matching < 0.6 

No comparator Imbalanced on 


The final security rating for this trial is 4 @ for the non-cognitive skills outcomes. 
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Appendix L: Cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention 
over three years. Cost ratings are awarded using the following criteria. 


| DY=X-Yeq a] o) de) a} 

Very low: less than £80 per pupil per year. 
EE Low: up to about £200 per pupil per year. 
Moderate: up to about £700 per pupil per year. 


EEEE High: up to £1,200 per pupil per year. 
[E£EEE Very high: over £1,200 per pupil per year. | 
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You may re-use this document/publication (not including logos) free of charge in any format or 
medium, under the terms of the Open Government Licence v2.0. 


To view this licence, visit www.nationalarchives.gov.uk/doc/open-government-icence/version/2 or 


email: psi@nationalarchives.gsi.gov.uk 


Where we have identified any third party copyright information you will need to obtain permission 
from the copyright holders concerned. The views expressed in this report are the authors’ and do 
not necessarily reflect those of the Department for Education. 


This document is available for download at www .educationendowmentfoundation.org.uk 
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